2024-09-15T03:48:30.714 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T03:48:30.736 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T03:48:30.820 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905333 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_latest} thrashers/sync workloads/pool-create-delete} email: skanta@redhat.com first_in_suite: false job_id: '7905333' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: filestore osd op queue: debug_random osd op queue cut off: debug_random osd sloppy crc: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - slow request - overall HEALTH_ sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 2126 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF+lXPJo6D5KxJc4ZEcTFLJnShYZGsoAQZ5560lCP61/8HPSEc+qrfLMHp6ht69e3JP4vEdqnNNEZ7UAArHI4eY= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFvpMn36bc2JmYB6aiw9Btz1iUb5Y59dE12hcrW26b6QKeWr74W6Hsg5dMUJU8lbRAdU+RTOLVOc6+j7j9pKW/o= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 - exec: client.0: - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_00:54:39 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T03:48:30.820 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-15T03:48:30.821 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-15T03:48:30.821 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T03:48:30.823 INFO:teuthology.task.internal:Checking packages... 2024-09-15T03:48:30.843 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-15T03:48:30.844 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T03:48:30.844 INFO:teuthology.packaging:ref: None 2024-09-15T03:48:30.844 INFO:teuthology.packaging:tag: None 2024-09-15T03:48:30.844 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T03:48:30.844 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:48:30.844 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T03:48:30.993 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643.gd75f2382 2024-09-15T03:48:30.995 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T03:48:31.004 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T03:48:31.004 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T03:48:31.021 INFO:teuthology.task.internal:Saving configuration 2024-09-15T03:48:31.036 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T03:48:31.045 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T03:48:31.071 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905333', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 03:43:32.850792', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF+lXPJo6D5KxJc4ZEcTFLJnShYZGsoAQZ5560lCP61/8HPSEc+qrfLMHp6ht69e3JP4vEdqnNNEZ7UAArHI4eY='} 2024-09-15T03:48:31.094 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905333', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 03:43:32.847581', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFvpMn36bc2JmYB6aiw9Btz1iUb5Y59dE12hcrW26b6QKeWr74W6Hsg5dMUJU8lbRAdU+RTOLVOc6+j7j9pKW/o='} 2024-09-15T03:48:31.095 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T03:48:31.107 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-15T03:48:31.107 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-15T03:48:31.107 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T03:48:31.173 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe98cf8fac0>, signals=[15]) 2024-09-15T03:48:31.174 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T03:48:31.182 INFO:teuthology.task.internal:Opening connections... 2024-09-15T03:48:31.183 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2024-09-15T03:48:31.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:48:31.263 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-09-15T03:48:31.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:48:31.343 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T03:48:31.352 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-09-15T03:48:31.370 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-09-15T03:48:31.371 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-09-15T03:48:31.428 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Stream" 2024-09-15T03:48:31.428 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="9" 2024-09-15T03:48:31.428 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos" 2024-09-15T03:48:31.428 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora" 2024-09-15T03:48:31.428 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="9" 2024-09-15T03:48:31.428 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el9" 2024-09-15T03:48:31.429 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T03:48:31.429 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2024-09-15T03:48:31.429 INFO:teuthology.orchestra.run.smithi110.stdout:LOGO="fedora-logo-icon" 2024-09-15T03:48:31.429 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T03:48:31.429 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/" 2024-09-15T03:48:31.429 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T03:48:31.429 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T03:48:31.429 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T03:48:31.430 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-09-15T03:48:31.454 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-09-15T03:48:31.473 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-09-15T03:48:31.474 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-09-15T03:48:31.531 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-09-15T03:48:31.531 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-09-15T03:48:31.531 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-09-15T03:48:31.531 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-09-15T03:48:31.531 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-09-15T03:48:31.531 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-09-15T03:48:31.531 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T03:48:31.531 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-09-15T03:48:31.531 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-09-15T03:48:31.532 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T03:48:31.532 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-09-15T03:48:31.532 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T03:48:31.532 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T03:48:31.532 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T03:48:31.532 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-09-15T03:48:31.554 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T03:48:31.563 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T03:48:31.573 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T03:48:31.574 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2024-09-15T03:48:31.576 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-09-15T03:48:31.592 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T03:48:31.601 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T03:48:31.602 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2024-09-15T03:48:31.634 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-09-15T03:48:31.738 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T03:48:31.794 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T03:48:31.809 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T03:48:31.809 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T03:48:31.809 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T03:48:31.810 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T03:48:31.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:48:31.827 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-09-15T03:48:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:48:31.828 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-09-15T03:48:31.886 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-503.el9.x86_64 2024-09-15T03:48:31.887 INFO:teuthology.task.kernel:Running kernel on smithi177: 5.14.0-503.el9.x86_64 2024-09-15T03:48:31.887 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum install -y kernel 2024-09-15T03:48:31.888 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-503.el9.x86_64 2024-09-15T03:48:31.888 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.14.0-503.el9.x86_64 2024-09-15T03:48:31.888 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum install -y kernel 2024-09-15T03:48:34.355 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-15T03:48:34.526 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 9.5 MB/s | 8.2 MB 00:00 2024-09-15T03:48:40.068 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-15T03:48:44.347 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 20 MB 00:08 2024-09-15T03:48:44.652 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-15T03:48:46.127 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 19 kB 00:00 2024-09-15T03:48:46.898 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-15T03:48:48.639 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-15T03:48:50.140 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 19 kB 00:00 2024-09-15T03:48:51.286 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-15T03:48:52.354 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-15T03:48:54.352 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T03:48:54.352 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T03:48:54.441 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:48:54.451 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:48:54.451 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-09-15T03:48:54.451 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:48:54.452 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:48:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T03:48:54.452 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-15T03:48:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T03:48:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T03:48:54.452 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T03:48:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:48:54.453 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:48:54.453 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:48:54.453 INFO:teuthology.orchestra.run.smithi177.stdout:Install 4 Packages 2024-09-15T03:48:54.453 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:48:54.453 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 79 M 2024-09-15T03:48:54.453 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 126 M 2024-09-15T03:48:54.454 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:48:54.556 INFO:teuthology.orchestra.run.smithi177.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 272 kB/s | 23 kB 00:00 2024-09-15T03:48:54.922 INFO:teuthology.orchestra.run.smithi177.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-09-15T03:48:55.306 INFO:teuthology.orchestra.run.smithi177.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 41 MB/s | 35 MB 00:00 2024-09-15T03:48:55.773 INFO:teuthology.orchestra.run.smithi177.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 23 MB/s | 29 MB 00:01 2024-09-15T03:48:55.773 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:48:55.773 INFO:teuthology.orchestra.run.smithi177.stdout:Total 60 MB/s | 79 MB 00:01 2024-09-15T03:48:56.461 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:48:56.700 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-15T03:48:56.806 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:48:56.807 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:48:57.794 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:48:57.794 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:48:58.715 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T03:48:58.715 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T03:48:58.800 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:48:58.807 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:48:58.807 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-09-15T03:48:58.807 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:48:58.807 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:48:58.807 INFO:teuthology.orchestra.run.smithi110.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T03:48:58.807 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-15T03:48:58.807 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T03:48:58.807 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T03:48:58.808 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T03:48:58.808 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:48:58.808 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:48:58.808 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:48:58.808 INFO:teuthology.orchestra.run.smithi110.stdout:Install 4 Packages 2024-09-15T03:48:58.808 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:48:58.809 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 79 M 2024-09-15T03:48:58.809 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 126 M 2024-09-15T03:48:58.809 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:48:58.935 INFO:teuthology.orchestra.run.smithi110.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 278 kB/s | 23 kB 00:00 2024-09-15T03:48:59.289 INFO:teuthology.orchestra.run.smithi110.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-09-15T03:48:59.677 INFO:teuthology.orchestra.run.smithi110.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 42 MB/s | 35 MB 00:00 2024-09-15T03:49:00.096 INFO:teuthology.orchestra.run.smithi110.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 25 MB/s | 29 MB 00:01 2024-09-15T03:49:00.097 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:49:00.097 INFO:teuthology.orchestra.run.smithi110.stdout:Total 61 MB/s | 79 MB 00:01 2024-09-15T03:49:00.258 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:49:00.779 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:49:02.488 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:49:02.488 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:49:02.488 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:49:02.488 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:49:02.488 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:49:02.489 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:49:03.794 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:49:03.918 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:49:04.529 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:49:05.238 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:49:05.321 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:49:08.111 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:49:08.240 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:49:08.735 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:49:08.777 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:49:13.087 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:49:13.131 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:49:13.657 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:49:18.033 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:49:51.938 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:49:51.978 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:49:52.324 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:49:52.324 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:49:52.324 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:49:52.324 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:49:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:49:53.158 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:49:53.158 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:49:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T03:49:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T03:49:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T03:49:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T03:49:53.159 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:49:53.159 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:49:53.531 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:49:53.660 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:49:53.660 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:49:53.660 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T03:49:53.660 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T03:49:53.660 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T03:49:53.660 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-15T03:49:53.661 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T03:49:53.661 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:49:53.720 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:49:53.720 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:49:53.720 DEBUG:teuthology.orchestra.run.smithi177:> sudo rpm -qi grub2-tools 2024-09-15T03:49:53.752 INFO:teuthology.orchestra.run.smithi177.stdout:Name : grub2-tools 2024-09-15T03:49:53.753 INFO:teuthology.orchestra.run.smithi177.stdout:Epoch : 1 2024-09-15T03:49:53.753 INFO:teuthology.orchestra.run.smithi177.stdout:Version : 2.06 2024-09-15T03:49:53.753 INFO:teuthology.orchestra.run.smithi177.stdout:Release : 82.el9 2024-09-15T03:49:53.753 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: x86_64 2024-09-15T03:49:53.753 INFO:teuthology.orchestra.run.smithi177.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T03:49:53.753 INFO:teuthology.orchestra.run.smithi177.stdout:Group : Unspecified 2024-09-15T03:49:53.753 INFO:teuthology.orchestra.run.smithi177.stdout:Size : 8274670 2024-09-15T03:49:53.753 INFO:teuthology.orchestra.run.smithi177.stdout:License : GPLv3+ 2024-09-15T03:49:53.753 INFO:teuthology.orchestra.run.smithi177.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T03:49:53.753 INFO:teuthology.orchestra.run.smithi177.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T03:49:53.753 INFO:teuthology.orchestra.run.smithi177.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T03:49:53.753 INFO:teuthology.orchestra.run.smithi177.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T03:49:53.753 INFO:teuthology.orchestra.run.smithi177.stdout:Packager : builder@centos.org 2024-09-15T03:49:53.754 INFO:teuthology.orchestra.run.smithi177.stdout:Vendor : CentOS 2024-09-15T03:49:53.754 INFO:teuthology.orchestra.run.smithi177.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T03:49:53.754 INFO:teuthology.orchestra.run.smithi177.stdout:Summary : Support tools for GRUB. 2024-09-15T03:49:53.754 INFO:teuthology.orchestra.run.smithi177.stdout:Description : 2024-09-15T03:49:53.754 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:49:53.754 INFO:teuthology.orchestra.run.smithi177.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T03:49:53.754 INFO:teuthology.orchestra.run.smithi177.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T03:49:53.754 INFO:teuthology.orchestra.run.smithi177.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T03:49:53.754 INFO:teuthology.orchestra.run.smithi177.stdout:hardware devices. 2024-09-15T03:49:53.754 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:49:53.754 INFO:teuthology.orchestra.run.smithi177.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T03:49:53.755 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T03:49:53.755 INFO:teuthology.task.kernel:Updating grub on smithi177 to boot 5.14.0-505.el9.x86_64 2024-09-15T03:49:53.755 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T03:49:54.403 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2024-09-15T03:49:55.663 INFO:teuthology.orchestra.run.smithi177.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T03:49:55.694 INFO:teuthology.orchestra.run.smithi177.stderr:done 2024-09-15T03:49:55.696 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-09-15T03:49:55.711 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.KBbg8FX7rX 2024-09-15T03:49:55.711 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KBbg8FX7rX 2024-09-15T03:49:55.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.KBbg8FX7rX 2024-09-15T03:49:55.921 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.KBbg8FX7rX is 6KB 2024-09-15T03:49:55.942 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.KBbg8FX7rX 2024-09-15T03:49:55.958 DEBUG:teuthology.orchestra.run.smithi177:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T03:49:56.022 INFO:teuthology.orchestra.run.smithi177.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T03:49:56.022 INFO:teuthology.orchestra.run.smithi177.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T03:49:56.022 INFO:teuthology.orchestra.run.smithi177.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T03:49:56.023 INFO:teuthology.orchestra.run.smithi177.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T03:49:56.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T03:49:56.115 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:49:56.154 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:49:56.186 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2024-09-15T03:49:56.549 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:49:56.549 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:49:56.550 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:49:56.550 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:49:57.379 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:49:57.379 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:49:57.379 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:49:57.379 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T03:49:57.379 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T03:49:57.379 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T03:49:57.379 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T03:49:57.379 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:49:57.380 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:49:57.766 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:49:57.902 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:49:57.902 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:49:57.902 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T03:49:57.902 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T03:49:57.902 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T03:49:57.902 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T03:49:57.902 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T03:49:57.903 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:49:57.962 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:49:57.962 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:49:57.962 DEBUG:teuthology.orchestra.run.smithi110:> sudo rpm -qi grub2-tools 2024-09-15T03:49:58.001 INFO:teuthology.orchestra.run.smithi110.stdout:Name : grub2-tools 2024-09-15T03:49:58.001 INFO:teuthology.orchestra.run.smithi110.stdout:Epoch : 1 2024-09-15T03:49:58.001 INFO:teuthology.orchestra.run.smithi110.stdout:Version : 2.06 2024-09-15T03:49:58.001 INFO:teuthology.orchestra.run.smithi110.stdout:Release : 82.el9 2024-09-15T03:49:58.001 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: x86_64 2024-09-15T03:49:58.001 INFO:teuthology.orchestra.run.smithi110.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T03:49:58.001 INFO:teuthology.orchestra.run.smithi110.stdout:Group : Unspecified 2024-09-15T03:49:58.001 INFO:teuthology.orchestra.run.smithi110.stdout:Size : 8274670 2024-09-15T03:49:58.001 INFO:teuthology.orchestra.run.smithi110.stdout:License : GPLv3+ 2024-09-15T03:49:58.002 INFO:teuthology.orchestra.run.smithi110.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T03:49:58.002 INFO:teuthology.orchestra.run.smithi110.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T03:49:58.002 INFO:teuthology.orchestra.run.smithi110.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T03:49:58.002 INFO:teuthology.orchestra.run.smithi110.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T03:49:58.002 INFO:teuthology.orchestra.run.smithi110.stdout:Packager : builder@centos.org 2024-09-15T03:49:58.002 INFO:teuthology.orchestra.run.smithi110.stdout:Vendor : CentOS 2024-09-15T03:49:58.002 INFO:teuthology.orchestra.run.smithi110.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T03:49:58.002 INFO:teuthology.orchestra.run.smithi110.stdout:Summary : Support tools for GRUB. 2024-09-15T03:49:58.002 INFO:teuthology.orchestra.run.smithi110.stdout:Description : 2024-09-15T03:49:58.002 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:49:58.002 INFO:teuthology.orchestra.run.smithi110.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T03:49:58.002 INFO:teuthology.orchestra.run.smithi110.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T03:49:58.002 INFO:teuthology.orchestra.run.smithi110.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T03:49:58.002 INFO:teuthology.orchestra.run.smithi110.stdout:hardware devices. 2024-09-15T03:49:58.003 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:49:58.003 INFO:teuthology.orchestra.run.smithi110.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T03:49:58.003 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T03:49:58.004 INFO:teuthology.task.kernel:Updating grub on smithi110 to boot 5.14.0-505.el9.x86_64 2024-09-15T03:49:58.004 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T03:49:58.673 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2024-09-15T03:50:00.033 INFO:teuthology.orchestra.run.smithi110.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T03:50:00.072 INFO:teuthology.orchestra.run.smithi110.stderr:done 2024-09-15T03:50:00.074 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-09-15T03:50:00.090 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.aTsZ34Kr1M 2024-09-15T03:50:00.090 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aTsZ34Kr1M 2024-09-15T03:50:00.164 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.aTsZ34Kr1M 2024-09-15T03:50:00.307 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.aTsZ34Kr1M is 6KB 2024-09-15T03:50:00.356 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.aTsZ34Kr1M 2024-09-15T03:50:00.371 DEBUG:teuthology.orchestra.run.smithi110:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T03:50:00.433 INFO:teuthology.orchestra.run.smithi110.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T03:50:00.433 INFO:teuthology.orchestra.run.smithi110.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T03:50:00.433 INFO:teuthology.orchestra.run.smithi110.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T03:50:00.433 INFO:teuthology.orchestra.run.smithi110.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T03:50:00.434 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T03:50:00.591 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2024-09-15T03:50:26.213 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-15T03:50:26.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-15T03:50:26.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:50:30.597 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T03:50:30.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-09-15T03:50:30.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:50:49.149 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2024-09-15T03:50:58.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-09-15T03:50:58.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:51:26.253 DEBUG:teuthology.orchestra.remote:timed out 2024-09-15T03:51:35.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-15T03:51:35.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:51:36.846 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-09-15T03:51:37.324 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-15T03:51:37.330 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-15T03:51:37.330 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-09-15T03:51:37.386 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-505.el9.x86_64 2024-09-15T03:51:37.386 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T03:51:37.386 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T03:51:37.386 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T03:51:38.387 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-15T03:51:38.387 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T03:51:38.466 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2024-09-15T03:51:38.486 DEBUG:teuthology.parallel:result is None 2024-09-15T03:51:58.160 DEBUG:teuthology.orchestra.remote:timed out 2024-09-15T03:52:10.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-09-15T03:52:10.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:52:10.543 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-09-15T03:52:10.942 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-09-15T03:52:10.942 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-15T03:52:10.942 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-09-15T03:52:10.959 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-505.el9.x86_64 2024-09-15T03:52:10.960 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T03:52:10.960 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T03:52:10.960 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T03:52:11.961 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T03:52:11.962 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T03:52:12.041 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2024-09-15T03:52:12.065 DEBUG:teuthology.parallel:result is None 2024-09-15T03:52:12.065 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T03:52:12.076 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T03:52:12.076 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T03:52:12.079 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T03:52:12.101 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T03:52:12.112 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T03:52:12.124 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T03:52:12.125 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T03:52:12.138 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T03:52:12.169 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T03:52:12.180 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T03:52:12.180 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T03:52:12.202 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T03:52:12.238 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:52:12.238 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:52:12.249 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:52:12.249 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:52:12.251 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T03:52:12.263 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T03:52:12.263 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T03:52:12.295 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T03:52:12.322 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T03:52:12.331 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T03:52:12.332 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T03:52:12.362 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T03:52:12.378 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:52:12.460 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:52:12.508 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:52:12.590 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:52:12.632 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:52:12.632 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T03:52:12.697 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:52:12.742 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:52:12.795 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:52:12.878 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:52:12.920 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:52:12.920 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T03:52:12.986 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2024-09-15T03:52:12.989 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-09-15T03:52:13.032 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:52:13.074 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:52:13.560 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T03:52:13.570 INFO:teuthology.task.internal:Starting timer... 2024-09-15T03:52:13.570 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T03:52:13.613 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T03:52:13.627 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2024-09-15T03:52:13.683 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: 2024-09-15T03:52:13.685 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-09-15T03:52:13.770 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-09-15T03:52:13.772 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T03:52:13.773 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2024-09-15T03:52:13.801 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2024-09-15T03:52:13.801 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-09-15T03:52:13.823 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-09-15T03:52:13.823 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi110.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2024-09-15T03:52:13.824 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:52:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:52:13.869 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:52:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:52:13.897 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T03:52:13.898 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2024-09-15T03:52:13.956 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-09-15T03:52:13.994 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T03:52:14.007 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T03:52:14.021 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T03:52:14.022 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi110.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T03:55:57.817 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi110.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-09-15T03:55:57.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-09-15T03:55:57.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:55:57.905 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-09-15T03:55:57.984 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-09-15T03:55:57.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-15T03:55:57.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:55:58.067 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-09-15T03:55:58.149 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-15T03:55:58.150 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T03:55:58.163 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T03:55:58.163 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T03:55:58.163 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:55:58.166 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T03:55:58.166 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:55:58.192 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T03:55:58.207 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T03:55:58.220 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T03:55:58.231 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2024-09-15T03:55:58.235 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T03:55:58.242 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2024-09-15T03:55:58.256 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T03:55:58.259 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-09-15T03:55:58.270 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-09-15T03:55:58.271 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T03:55:58.285 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T03:55:58.299 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T03:55:58.337 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:55:58.340 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:55:58.341 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-09-15T03:55:58.341 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:55:58.341 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:55:58.341 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:55:58.341 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:55:58.354 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:55:58.358 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:55:58.358 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-09-15T03:55:58.358 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:55:58.358 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:55:58.358 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:55:58.358 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:55:58.358 INFO:teuthology.run_tasks:Running task install... 2024-09-15T03:55:58.366 DEBUG:teuthology.task.install:project ceph 2024-09-15T03:55:58.366 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T03:55:58.366 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-15T03:55:58.367 INFO:teuthology.task.install:Using flavor: default 2024-09-15T03:55:58.372 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T03:55:58.373 INFO:teuthology.task.install:extra packages: [] 2024-09-15T03:55:58.373 DEBUG:teuthology.orchestra.run.smithi110:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T03:55:58.374 DEBUG:teuthology.orchestra.run.smithi177:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T03:55:58.935 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:44 ago on Sun 15 Sep 2024 03:55:14 AM UTC. 2024-09-15T03:55:58.957 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:44 ago on Sun 15 Sep 2024 03:55:14 AM UTC. 2024-09-15T03:55:59.022 INFO:teuthology.orchestra.run.smithi110.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T03:55:59.045 INFO:teuthology.orchestra.run.smithi177.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T03:55:59.064 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:55:59.065 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-15T03:55:59.065 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:55:59.087 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:55:59.088 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-15T03:55:59.088 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:55:59.134 DEBUG:teuthology.orchestra.run.smithi110:> sudo dnf -y copr enable ceph/el9 2024-09-15T03:55:59.148 DEBUG:teuthology.orchestra.run.smithi177:> sudo dnf -y copr enable ceph/el9 2024-09-15T03:55:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T03:55:59.304 INFO:teuthology.orchestra.run.smithi110.stderr:of the main distribution, and quality may vary. 2024-09-15T03:55:59.304 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:55:59.305 INFO:teuthology.orchestra.run.smithi110.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T03:55:59.305 INFO:teuthology.orchestra.run.smithi110.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T03:55:59.305 INFO:teuthology.orchestra.run.smithi110.stderr:, 2024-09-15T03:55:59.305 INFO:teuthology.orchestra.run.smithi110.stderr:and packages are not held to any quality or security level. 2024-09-15T03:55:59.305 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:55:59.305 INFO:teuthology.orchestra.run.smithi110.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T03:55:59.306 INFO:teuthology.orchestra.run.smithi110.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T03:55:59.319 INFO:teuthology.orchestra.run.smithi177.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T03:55:59.320 INFO:teuthology.orchestra.run.smithi177.stderr:of the main distribution, and quality may vary. 2024-09-15T03:55:59.320 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:55:59.320 INFO:teuthology.orchestra.run.smithi177.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T03:55:59.320 INFO:teuthology.orchestra.run.smithi177.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T03:55:59.320 INFO:teuthology.orchestra.run.smithi177.stderr:, 2024-09-15T03:55:59.321 INFO:teuthology.orchestra.run.smithi177.stderr:and packages are not held to any quality or security level. 2024-09-15T03:55:59.321 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:55:59.321 INFO:teuthology.orchestra.run.smithi177.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T03:55:59.321 INFO:teuthology.orchestra.run.smithi177.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T03:55:59.357 INFO:teuthology.orchestra.run.smithi110.stdout:Repository successfully enabled. 2024-09-15T03:55:59.371 INFO:teuthology.orchestra.run.smithi177.stdout:Repository successfully enabled. 2024-09-15T03:55:59.392 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T03:55:59.393 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:55:59.408 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T03:55:59.408 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:55:59.545 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T03:55:59.545 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T03:55:59.554 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T03:55:59.554 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T03:55:59.658 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T03:55:59.658 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:55:59.658 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T03:55:59.666 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T03:55:59.667 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:55:59.667 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T03:55:59.683 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T03:55:59.684 DEBUG:teuthology.orchestra.run.smithi110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T03:55:59.692 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T03:55:59.692 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T03:55:59.749 DEBUG:teuthology.orchestra.run.smithi110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T03:55:59.757 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T03:55:59.824 DEBUG:teuthology.orchestra.run.smithi110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T03:55:59.832 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T03:55:59.888 INFO:teuthology.orchestra.run.smithi110.stdout:check_obsoletes = 1 2024-09-15T03:55:59.890 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum clean all 2024-09-15T03:55:59.896 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-09-15T03:55:59.898 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-09-15T03:56:00.122 INFO:teuthology.orchestra.run.smithi177.stdout:45 files removed 2024-09-15T03:56:00.125 INFO:teuthology.orchestra.run.smithi110.stdout:45 files removed 2024-09-15T03:56:00.150 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw 2024-09-15T03:56:00.151 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw 2024-09-15T03:56:00.706 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 284 kB/s | 77 kB 00:00 2024-09-15T03:56:00.707 INFO:teuthology.orchestra.run.smithi110.stdout:ceph packages for x86_64 286 kB/s | 77 kB 00:00 2024-09-15T03:56:00.990 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-15T03:56:00.999 INFO:teuthology.orchestra.run.smithi110.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-09-15T03:56:01.257 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 7.8 kB/s | 1.8 kB 00:00 2024-09-15T03:56:01.275 INFO:teuthology.orchestra.run.smithi110.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-15T03:56:01.458 INFO:teuthology.orchestra.run.smithi177.stdout:Copr repo for el9 owned by ceph 46 kB/s | 7.6 kB 00:00 2024-09-15T03:56:01.525 INFO:teuthology.orchestra.run.smithi110.stdout:Copr repo for el9 owned by ceph 35 kB/s | 7.6 kB 00:00 2024-09-15T03:56:02.058 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-15T03:56:02.934 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 8.2 MB 00:01 2024-09-15T03:56:04.809 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-09-15T03:56:05.377 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 15 MB/s | 20 MB 00:01 2024-09-15T03:56:09.571 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-15T03:56:10.401 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-15T03:56:11.287 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 19 kB 00:00 2024-09-15T03:56:11.826 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 104 kB/s | 19 kB 00:00 2024-09-15T03:56:12.267 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-15T03:56:12.806 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-15T03:56:17.623 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-15T03:56:18.261 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-15T03:56:20.099 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:56:20.117 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:56:20.117 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-15T03:56:20.117 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:56:20.117 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:56:20.118 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T03:56:20.118 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrading: 2024-09-15T03:56:20.118 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T03:56:20.118 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:56:20.118 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-15T03:56:20.118 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T03:56:20.118 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T03:56:20.119 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:56:20.119 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T03:56:20.119 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T03:56:20.119 INFO:teuthology.orchestra.run.smithi110.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T03:56:20.119 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T03:56:20.119 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T03:56:20.119 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T03:56:20.119 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T03:56:20.119 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T03:56:20.120 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T03:56:20.120 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T03:56:20.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T03:56:20.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T03:56:20.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T03:56:20.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T03:56:20.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T03:56:20.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T03:56:20.120 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T03:56:20.121 INFO:teuthology.orchestra.run.smithi110.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T03:56:20.121 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:56:20.121 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:56:20.121 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:56:20.121 INFO:teuthology.orchestra.run.smithi110.stdout:Install 22 Packages 2024-09-15T03:56:20.121 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrade 2 Packages 2024-09-15T03:56:20.121 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:56:20.122 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 58 M 2024-09-15T03:56:20.122 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:56:20.721 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:56:20.729 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:56:20.729 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-15T03:56:20.729 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:56:20.729 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:56:20.729 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T03:56:20.729 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-09-15T03:56:20.729 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T03:56:20.729 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:56:20.729 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-15T03:56:20.729 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T03:56:20.729 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T03:56:20.729 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:56:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T03:56:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T03:56:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T03:56:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T03:56:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T03:56:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T03:56:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T03:56:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T03:56:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T03:56:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T03:56:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T03:56:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T03:56:20.730 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T03:56:20.731 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T03:56:20.731 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T03:56:20.731 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T03:56:20.731 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T03:56:20.731 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T03:56:20.731 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:56:20.731 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:56:20.731 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:56:20.731 INFO:teuthology.orchestra.run.smithi177.stdout:Install 22 Packages 2024-09-15T03:56:20.731 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 2 Packages 2024-09-15T03:56:20.731 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:56:20.732 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 58 M 2024-09-15T03:56:20.732 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:56:23.045 INFO:teuthology.orchestra.run.smithi177.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 2.2 MB/s | 5.0 MB 00:02 2024-09-15T03:56:23.057 INFO:teuthology.orchestra.run.smithi110.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 1.9 MB/s | 5.0 MB 00:02 2024-09-15T03:56:23.146 INFO:teuthology.orchestra.run.smithi177.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 252 kB/s | 25 kB 00:00 2024-09-15T03:56:23.157 INFO:teuthology.orchestra.run.smithi110.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 252 kB/s | 25 kB 00:00 2024-09-15T03:56:23.488 INFO:teuthology.orchestra.run.smithi177.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 1.8 MB/s | 637 kB 00:00 2024-09-15T03:56:23.549 INFO:teuthology.orchestra.run.smithi110.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 1.6 MB/s | 637 kB 00:00 2024-09-15T03:56:23.654 INFO:teuthology.orchestra.run.smithi177.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 2.7 MB/s | 458 kB 00:00 2024-09-15T03:56:23.858 INFO:teuthology.orchestra.run.smithi110.stdout:(4/24): libradosstriper1-17.2.7-1643.gd75f2382. 1.4 MB/s | 458 kB 00:00 2024-09-15T03:56:25.038 INFO:teuthology.orchestra.run.smithi177.stdout:(5/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 3.6 MB/s | 4.9 MB 00:01 2024-09-15T03:56:25.138 INFO:teuthology.orchestra.run.smithi177.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 452 kB/s | 45 kB 00:00 2024-09-15T03:56:25.238 INFO:teuthology.orchestra.run.smithi177.stdout:(7/24): python3-ceph-common-17.2.7-1643.gd75f23 1.2 MB/s | 120 kB 00:00 2024-09-15T03:56:25.339 INFO:teuthology.orchestra.run.smithi177.stdout:(8/24): python3-cephfs-17.2.7-1643.gd75f2382.el 1.5 MB/s | 156 kB 00:00 2024-09-15T03:56:25.489 INFO:teuthology.orchestra.run.smithi177.stdout:(9/24): python3-rados-17.2.7-1643.gd75f2382.el9 2.1 MB/s | 320 kB 00:00 2024-09-15T03:56:25.581 INFO:teuthology.orchestra.run.smithi177.stdout:(10/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 3.1 MB/s | 295 kB 00:00 2024-09-15T03:56:25.664 INFO:teuthology.orchestra.run.smithi177.stdout:(11/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.2 MB/s | 100 kB 00:00 2024-09-15T03:56:25.781 INFO:teuthology.orchestra.run.smithi177.stdout:(12/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-09-15T03:56:25.823 INFO:teuthology.orchestra.run.smithi177.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-15T03:56:25.898 INFO:teuthology.orchestra.run.smithi177.stdout:(14/24): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-09-15T03:56:25.940 INFO:teuthology.orchestra.run.smithi177.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.9 MB/s | 292 kB 00:00 2024-09-15T03:56:25.982 INFO:teuthology.orchestra.run.smithi177.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-15T03:56:26.033 INFO:teuthology.orchestra.run.smithi177.stdout:(17/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-15T03:56:26.058 INFO:teuthology.orchestra.run.smithi177.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-15T03:56:26.083 INFO:teuthology.orchestra.run.smithi177.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-15T03:56:26.144 INFO:teuthology.orchestra.run.smithi177.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 26 MB/s | 1.6 MB 00:00 2024-09-15T03:56:26.175 INFO:teuthology.orchestra.run.smithi110.stdout:(5/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 2.1 MB/s | 4.9 MB 00:02 2024-09-15T03:56:26.259 INFO:teuthology.orchestra.run.smithi110.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 542 kB/s | 45 kB 00:00 2024-09-15T03:56:26.403 INFO:teuthology.orchestra.run.smithi177.stdout:(21/24): ceph-common-17.2.7-1643.gd75f2382.el9. 4.0 MB/s | 22 MB 00:05 2024-09-15T03:56:26.476 INFO:teuthology.orchestra.run.smithi110.stdout:(7/24): ceph-common-17.2.7-1643.gd75f2382.el9.x 3.7 MB/s | 22 MB 00:06 2024-09-15T03:56:26.509 INFO:teuthology.orchestra.run.smithi110.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 480 kB/s | 120 kB 00:00 2024-09-15T03:56:26.559 INFO:teuthology.orchestra.run.smithi110.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 1.8 MB/s | 156 kB 00:00 2024-09-15T03:56:26.660 INFO:teuthology.orchestra.run.smithi110.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 2.1 MB/s | 320 kB 00:00 2024-09-15T03:56:26.702 INFO:teuthology.orchestra.run.smithi110.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 2.0 MB/s | 295 kB 00:00 2024-09-15T03:56:26.744 INFO:teuthology.orchestra.run.smithi110.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.2 MB/s | 100 kB 00:00 2024-09-15T03:56:26.836 INFO:teuthology.orchestra.run.smithi110.stdout:(13/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-15T03:56:26.853 INFO:teuthology.orchestra.run.smithi177.stdout:(22/24): librados2-17.2.7-1643.gd75f2382.el9.x8 4.3 MB/s | 3.1 MB 00:00 2024-09-15T03:56:26.861 INFO:teuthology.orchestra.run.smithi110.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-09-15T03:56:26.903 INFO:teuthology.orchestra.run.smithi177.stdout:(23/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 5.9 MB/s | 2.9 MB 00:00 2024-09-15T03:56:26.945 INFO:teuthology.orchestra.run.smithi110.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-09-15T03:56:26.970 INFO:teuthology.orchestra.run.smithi110.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-15T03:56:27.020 INFO:teuthology.orchestra.run.smithi110.stdout:(17/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-15T03:56:27.046 INFO:teuthology.orchestra.run.smithi110.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 420 kB/s | 42 kB 00:00 2024-09-15T03:56:27.071 INFO:teuthology.orchestra.run.smithi110.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-09-15T03:56:27.121 INFO:teuthology.orchestra.run.smithi110.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 896 kB/s | 67 kB 00:00 2024-09-15T03:56:27.163 INFO:teuthology.orchestra.run.smithi110.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-09-15T03:56:27.588 INFO:teuthology.orchestra.run.smithi110.stdout:(22/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 6.9 MB/s | 2.9 MB 00:00 2024-09-15T03:56:27.822 INFO:teuthology.orchestra.run.smithi110.stdout:(23/24): librados2-17.2.7-1643.gd75f2382.el9.x8 4.4 MB/s | 3.1 MB 00:00 2024-09-15T03:56:27.903 INFO:teuthology.orchestra.run.smithi177.stdout:(24/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9 2.0 MB/s | 14 MB 00:07 2024-09-15T03:56:27.907 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:56:27.907 INFO:teuthology.orchestra.run.smithi177.stdout:Total 8.1 MB/s | 58 MB 00:07 2024-09-15T03:56:27.973 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:56:28.006 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:56:28.006 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:56:28.339 INFO:teuthology.orchestra.run.smithi110.stdout:(24/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9 1.8 MB/s | 14 MB 00:07 2024-09-15T03:56:28.344 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:56:28.344 INFO:teuthology.orchestra.run.smithi110.stdout:Total 7.1 MB/s | 58 MB 00:08 2024-09-15T03:56:28.405 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:56:28.439 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:56:28.439 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:56:28.660 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:56:28.660 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:56:29.102 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:56:29.103 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:56:29.891 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:56:30.225 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T03:56:30.337 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:56:30.338 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:56:30.431 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:56:30.518 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:56:30.548 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:56:30.637 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:56:30.682 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T03:56:30.703 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T03:56:30.711 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T03:56:30.783 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:56:30.875 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:56:30.961 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:56:30.984 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:56:31.073 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:56:31.121 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T03:56:31.149 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T03:56:31.164 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T03:56:31.180 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:56:31.247 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:56:31.276 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T03:56:31.318 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T03:56:31.356 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:56:31.439 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:56:31.470 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:56:31.535 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:56:31.584 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T03:56:31.599 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T03:56:31.620 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T03:56:31.621 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:56:31.645 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T03:56:31.684 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:56:31.721 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T03:56:31.772 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T03:56:31.818 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:56:31.880 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:56:31.907 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:56:31.938 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T03:56:31.973 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:56:31.983 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:56:32.023 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T03:56:32.066 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T03:56:32.072 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:56:32.100 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T03:56:32.147 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T03:56:32.177 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T03:56:32.391 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T03:56:32.436 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:56:32.527 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:56:32.600 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T03:56:32.630 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T03:56:34.602 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:56:34.639 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:56:35.042 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:56:35.087 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:56:35.167 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:56:35.201 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:56:35.307 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:56:35.307 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T03:56:35.307 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T03:56:35.308 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:56:35.344 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:56:35.606 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:56:35.648 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:56:35.751 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:56:35.752 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T03:56:35.752 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T03:56:35.752 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:56:35.783 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:56:44.855 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:56:44.855 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-09-15T03:56:44.855 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-09-15T03:56:44.855 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-09-15T03:56:44.856 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-09-15T03:56:44.856 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-09-15T03:56:44.856 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2024-09-15T03:56:44.856 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-09-15T03:56:44.856 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:56:45.415 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:56:45.415 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /sys 2024-09-15T03:56:45.415 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /proc 2024-09-15T03:56:45.415 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /mnt 2024-09-15T03:56:45.415 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /var/tmp 2024-09-15T03:56:45.415 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /home 2024-09-15T03:56:45.415 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /root 2024-09-15T03:56:45.415 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /tmp 2024-09-15T03:56:45.415 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:56:46.150 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:56:46.236 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:56:46.236 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:56:46.236 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T03:56:46.236 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:56:46.236 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:56:46.236 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:56:46.238 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:56:46.336 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:56:46.336 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:56:46.739 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:56:46.809 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:56:46.809 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:56:46.809 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T03:56:46.809 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:56:46.809 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:56:46.810 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:56:46.811 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:56:46.900 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:56:46.901 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:56:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:56:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T03:56:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:56:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:56:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:56:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T03:56:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T03:56:47.628 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T03:56:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T03:56:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T03:56:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T03:56:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:56:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:56:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T03:56:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T03:56:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T03:56:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T03:56:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T03:56:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T03:56:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T03:56:47.629 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T03:56:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T03:56:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T03:56:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:56:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T03:56:47.630 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T03:56:48.175 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:56:48.175 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T03:56:48.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:56:48.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:56:48.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:56:48.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T03:56:48.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T03:56:48.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T03:56:48.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T03:56:48.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T03:56:48.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T03:56:48.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:56:48.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:56:48.176 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T03:56:48.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T03:56:48.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T03:56:48.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T03:56:48.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T03:56:48.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T03:56:48.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T03:56:48.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T03:56:48.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T03:56:48.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T03:56:48.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:56:48.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T03:56:48.177 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T03:56:48.685 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:56:48.685 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:56:48.685 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-09-15T03:56:48.685 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.685 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.685 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:56:48.685 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.685 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.685 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.685 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.685 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T03:56:48.686 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.686 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T03:56:48.686 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T03:56:48.686 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T03:56:48.686 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.686 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T03:56:48.686 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.686 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T03:56:48.686 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T03:56:48.686 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.686 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.686 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.686 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T03:56:48.686 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.687 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.687 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:48.687 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T03:56:48.687 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:56:48.687 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:56:49.090 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-test 2024-09-15T03:56:49.184 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:56:49.184 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:56:49.184 INFO:teuthology.orchestra.run.smithi110.stdout:Upgraded: 2024-09-15T03:56:49.184 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.184 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.184 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:56:49.184 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.184 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.185 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.185 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.185 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T03:56:49.185 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.185 INFO:teuthology.orchestra.run.smithi110.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T03:56:49.185 INFO:teuthology.orchestra.run.smithi110.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T03:56:49.185 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T03:56:49.185 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.185 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T03:56:49.185 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.185 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T03:56:49.185 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T03:56:49.185 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.185 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.186 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.186 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T03:56:49.186 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.186 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.186 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:56:49.186 INFO:teuthology.orchestra.run.smithi110.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T03:56:49.186 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:56:49.186 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:56:49.587 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-test 2024-09-15T03:56:49.653 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:31 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:56:49.786 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:56:49.787 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:56:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-15T03:56:49.787 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:56:49.787 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:56:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T03:56:49.787 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-15T03:56:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T03:56:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T03:56:49.787 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T03:56:49.788 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:56:49.788 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:56:49.788 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:56:49.788 INFO:teuthology.orchestra.run.smithi177.stdout:Install 4 Packages 2024-09-15T03:56:49.788 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:56:49.788 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 48 M 2024-09-15T03:56:49.789 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 193 M 2024-09-15T03:56:49.789 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:56:50.027 INFO:teuthology.orchestra.run.smithi177.stdout:(1/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 753 kB/s | 163 kB 00:00 2024-09-15T03:56:50.052 INFO:teuthology.orchestra.run.smithi177.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-15T03:56:50.150 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:33 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:56:50.169 INFO:teuthology.orchestra.run.smithi177.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 449 kB/s | 64 kB 00:00 2024-09-15T03:56:50.289 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:56:50.290 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:56:50.290 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-15T03:56:50.290 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:56:50.290 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:56:50.290 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T03:56:50.291 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-15T03:56:50.291 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T03:56:50.291 INFO:teuthology.orchestra.run.smithi110.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T03:56:50.291 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T03:56:50.291 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:56:50.291 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:56:50.291 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:56:50.291 INFO:teuthology.orchestra.run.smithi110.stdout:Install 4 Packages 2024-09-15T03:56:50.291 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:56:50.292 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 48 M 2024-09-15T03:56:50.292 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 193 M 2024-09-15T03:56:50.292 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:56:50.565 INFO:teuthology.orchestra.run.smithi110.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-15T03:56:50.590 INFO:teuthology.orchestra.run.smithi110.stdout:(2/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 590 kB/s | 163 kB 00:00 2024-09-15T03:56:50.615 INFO:teuthology.orchestra.run.smithi110.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-15T03:56:52.595 INFO:teuthology.orchestra.run.smithi177.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 17 MB/s | 48 MB 00:02 2024-09-15T03:56:52.596 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:56:52.597 INFO:teuthology.orchestra.run.smithi177.stdout:Total 17 MB/s | 48 MB 00:02 2024-09-15T03:56:52.625 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:56:52.644 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:56:52.644 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:56:53.009 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:56:53.009 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:56:53.537 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:56:53.629 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T03:56:53.669 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T03:56:53.696 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:56:56.940 INFO:teuthology.orchestra.run.smithi110.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 7.2 MB/s | 48 MB 00:06 2024-09-15T03:56:56.941 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:56:56.942 INFO:teuthology.orchestra.run.smithi110.stdout:Total 7.3 MB/s | 48 MB 00:06 2024-09-15T03:56:56.967 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:56:56.985 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:56:56.986 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:56:57.350 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:56:57.350 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:56:57.888 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:56:57.960 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T03:56:58.006 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T03:56:58.034 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:56:58.119 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:56:58.194 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:57:00.465 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:57:00.465 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T03:57:00.465 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T03:57:00.465 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T03:57:00.790 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T03:57:00.791 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:00.791 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:57:00.791 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:00.791 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:00.791 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T03:57:00.791 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T03:57:00.791 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:00.791 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:01.078 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph 2024-09-15T03:57:01.644 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:43 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:01.785 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:01.788 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-15T03:57:01.788 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:01.788 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:57:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T03:57:01.788 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-15T03:57:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T03:57:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T03:57:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T03:57:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T03:57:01.788 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T03:57:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T03:57:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T03:57:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T03:57:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T03:57:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T03:57:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T03:57:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T03:57:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections 2024-09-15T03:57:01.789 INFO:teuthology.orchestra.run.smithi177.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T03:57:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T03:57:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T03:57:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T03:57:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T03:57:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T03:57:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T03:57:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T03:57:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T03:57:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T03:57:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T03:57:01.790 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T03:57:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T03:57:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T03:57:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T03:57:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T03:57:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T03:57:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T03:57:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T03:57:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T03:57:01.791 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T03:57:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:01.792 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:57:01.792 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:01.792 INFO:teuthology.orchestra.run.smithi177.stdout:Install 34 Packages 2024-09-15T03:57:01.792 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:01.793 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 27 M 2024-09-15T03:57:01.793 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 96 M 2024-09-15T03:57:01.793 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:57:02.111 INFO:teuthology.orchestra.run.smithi177.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 44 kB/s | 6.5 kB 00:00 2024-09-15T03:57:02.270 INFO:teuthology.orchestra.run.smithi177.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 4.6 MB/s | 1.4 MB 00:00 2024-09-15T03:57:02.312 INFO:teuthology.orchestra.run.smithi177.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 6.0 MB/s | 2.1 MB 00:00 2024-09-15T03:57:02.370 INFO:teuthology.orchestra.run.smithi177.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 4.0 MB/s | 240 kB 00:00 2024-09-15T03:57:02.443 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:57:02.462 INFO:teuthology.orchestra.run.smithi177.stdout:(5/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 11 MB/s | 3.9 MB 00:00 2024-09-15T03:57:02.514 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:57:02.596 INFO:teuthology.orchestra.run.smithi177.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-15T03:57:02.771 INFO:teuthology.orchestra.run.smithi177.stdout:(7/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 31 MB/s | 15 MB 00:00 2024-09-15T03:57:02.796 INFO:teuthology.orchestra.run.smithi177.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 532 kB/s | 106 kB 00:00 2024-09-15T03:57:02.847 INFO:teuthology.orchestra.run.smithi177.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-15T03:57:02.905 INFO:teuthology.orchestra.run.smithi177.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-15T03:57:02.939 INFO:teuthology.orchestra.run.smithi177.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 2.6 MB/s | 1.2 MB 00:00 2024-09-15T03:57:02.964 INFO:teuthology.orchestra.run.smithi177.stdout:(12/34): python3-pycparser-2.20-6.el9.noarch.rp 702 kB/s | 135 kB 00:00 2024-09-15T03:57:02.989 INFO:teuthology.orchestra.run.smithi177.stdout:(13/34): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-15T03:57:03.015 INFO:teuthology.orchestra.run.smithi177.stdout:(14/34): python3-markupsafe-1.1.1-12.el9.x86_64 460 kB/s | 35 kB 00:00 2024-09-15T03:57:03.040 INFO:teuthology.orchestra.run.smithi177.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-15T03:57:03.065 INFO:teuthology.orchestra.run.smithi177.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 675 kB/s | 51 kB 00:00 2024-09-15T03:57:03.091 INFO:teuthology.orchestra.run.smithi177.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-15T03:57:03.132 INFO:teuthology.orchestra.run.smithi177.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 472 kB/s | 43 kB 00:00 2024-09-15T03:57:03.158 INFO:teuthology.orchestra.run.smithi177.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 456 kB/s | 11 kB 00:00 2024-09-15T03:57:03.183 INFO:teuthology.orchestra.run.smithi177.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-15T03:57:03.200 INFO:teuthology.orchestra.run.smithi177.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 421 kB/s | 18 kB 00:00 2024-09-15T03:57:03.217 INFO:teuthology.orchestra.run.smithi177.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 688 kB/s | 23 kB 00:00 2024-09-15T03:57:03.254 INFO:teuthology.orchestra.run.smithi177.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 362 kB/s | 19 kB 00:00 2024-09-15T03:57:03.279 INFO:teuthology.orchestra.run.smithi177.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 319 kB/s | 20 kB 00:00 2024-09-15T03:57:03.304 INFO:teuthology.orchestra.run.smithi177.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 928 kB/s | 46 kB 00:00 2024-09-15T03:57:03.330 INFO:teuthology.orchestra.run.smithi177.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-15T03:57:03.355 INFO:teuthology.orchestra.run.smithi177.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-15T03:57:03.380 INFO:teuthology.orchestra.run.smithi177.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-15T03:57:03.397 INFO:teuthology.orchestra.run.smithi177.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-15T03:57:03.422 INFO:teuthology.orchestra.run.smithi177.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 853 kB/s | 36 kB 00:00 2024-09-15T03:57:03.447 INFO:teuthology.orchestra.run.smithi177.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-15T03:57:03.473 INFO:teuthology.orchestra.run.smithi177.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 794 kB/s | 20 kB 00:00 2024-09-15T03:57:03.498 INFO:teuthology.orchestra.run.smithi177.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-15T03:57:03.523 INFO:teuthology.orchestra.run.smithi177.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-15T03:57:03.529 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:03.529 INFO:teuthology.orchestra.run.smithi177.stdout:Total 16 MB/s | 27 MB 00:01 2024-09-15T03:57:03.653 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:57:03.690 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:57:03.690 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:57:04.093 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:57:04.093 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:57:04.684 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:57:04.684 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T03:57:04.684 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T03:57:04.684 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T03:57:05.002 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T03:57:05.003 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:05.003 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:57:05.003 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:05.003 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:05.003 INFO:teuthology.orchestra.run.smithi110.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T03:57:05.003 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T03:57:05.003 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:05.003 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:57:05.318 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph 2024-09-15T03:57:05.328 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:57:05.401 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T03:57:05.442 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T03:57:05.475 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T03:57:05.566 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T03:57:05.644 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T03:57:05.686 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T03:57:05.729 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T03:57:05.759 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T03:57:05.793 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T03:57:05.827 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T03:57:05.867 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T03:57:05.902 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T03:57:05.903 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:48 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:57:05.937 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T03:57:05.985 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T03:57:06.026 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:57:06.045 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:57:06.047 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:06.047 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-15T03:57:06.047 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:06.047 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:57:06.047 INFO:teuthology.orchestra.run.smithi110.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T03:57:06.048 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-15T03:57:06.048 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T03:57:06.048 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T03:57:06.048 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T03:57:06.048 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T03:57:06.048 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T03:57:06.048 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T03:57:06.048 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T03:57:06.048 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T03:57:06.048 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T03:57:06.048 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T03:57:06.049 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T03:57:06.049 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T03:57:06.049 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections 2024-09-15T03:57:06.049 INFO:teuthology.orchestra.run.smithi110.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T03:57:06.049 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T03:57:06.049 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T03:57:06.049 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T03:57:06.049 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T03:57:06.049 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T03:57:06.049 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T03:57:06.049 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T03:57:06.049 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T03:57:06.050 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T03:57:06.050 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T03:57:06.050 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T03:57:06.050 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T03:57:06.050 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T03:57:06.050 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T03:57:06.050 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T03:57:06.050 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T03:57:06.050 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T03:57:06.050 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T03:57:06.050 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T03:57:06.050 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T03:57:06.050 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:06.051 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:57:06.051 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:06.051 INFO:teuthology.orchestra.run.smithi110.stdout:Install 34 Packages 2024-09-15T03:57:06.051 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:06.051 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 27 M 2024-09-15T03:57:06.051 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 96 M 2024-09-15T03:57:06.051 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:57:06.099 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T03:57:06.236 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T03:57:06.317 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T03:57:06.370 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T03:57:06.408 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T03:57:06.457 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T03:57:06.521 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T03:57:06.709 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T03:57:06.717 INFO:teuthology.orchestra.run.smithi110.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 46 kB/s | 6.5 kB 00:00 2024-09-15T03:57:06.755 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T03:57:06.840 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T03:57:06.859 INFO:teuthology.orchestra.run.smithi110.stdout:(2/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 5.0 MB/s | 1.4 MB 00:00 2024-09-15T03:57:06.929 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T03:57:06.951 INFO:teuthology.orchestra.run.smithi110.stdout:(3/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 5.5 MB/s | 2.1 MB 00:00 2024-09-15T03:57:06.962 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T03:57:07.043 INFO:teuthology.orchestra.run.smithi110.stdout:(4/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 12 MB/s | 3.9 MB 00:00 2024-09-15T03:57:07.076 INFO:teuthology.orchestra.run.smithi110.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 1.9 MB/s | 240 kB 00:00 2024-09-15T03:57:07.077 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T03:57:07.218 INFO:teuthology.orchestra.run.smithi110.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-15T03:57:07.224 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T03:57:07.256 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:57:07.283 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:57:07.283 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:57:07.283 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T03:57:07.283 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:57:07.283 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:57:07.283 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:07.385 INFO:teuthology.orchestra.run.smithi110.stdout:(7/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86_ 29 MB/s | 15 MB 00:00 2024-09-15T03:57:07.411 INFO:teuthology.orchestra.run.smithi110.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 555 kB/s | 106 kB 00:00 2024-09-15T03:57:07.461 INFO:teuthology.orchestra.run.smithi110.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-15T03:57:07.503 INFO:teuthology.orchestra.run.smithi110.stdout:(10/34): python3-cryptography-36.0.1-4.el9.x86_ 2.9 MB/s | 1.2 MB 00:00 2024-09-15T03:57:07.528 INFO:teuthology.orchestra.run.smithi110.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-15T03:57:07.578 INFO:teuthology.orchestra.run.smithi110.stdout:(12/34): python3-pycparser-2.20-6.el9.noarch.rp 702 kB/s | 135 kB 00:00 2024-09-15T03:57:07.629 INFO:teuthology.orchestra.run.smithi110.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 346 kB/s | 35 kB 00:00 2024-09-15T03:57:07.670 INFO:teuthology.orchestra.run.smithi110.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-15T03:57:07.704 INFO:teuthology.orchestra.run.smithi110.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-09-15T03:57:07.730 INFO:teuthology.orchestra.run.smithi110.stdout:(16/34): python3-packaging-20.9-5.el9.noarch.rp 511 kB/s | 77 kB 00:00 2024-09-15T03:57:07.755 INFO:teuthology.orchestra.run.smithi110.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 495 kB/s | 42 kB 00:00 2024-09-15T03:57:07.805 INFO:teuthology.orchestra.run.smithi110.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-15T03:57:07.830 INFO:teuthology.orchestra.run.smithi110.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 465 kB/s | 11 kB 00:00 2024-09-15T03:57:07.856 INFO:teuthology.orchestra.run.smithi110.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-15T03:57:07.873 INFO:teuthology.orchestra.run.smithi110.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 422 kB/s | 18 kB 00:00 2024-09-15T03:57:07.898 INFO:teuthology.orchestra.run.smithi110.stdout:(22/34): python3-jaraco-functools-3.5.0-2.el9.n 780 kB/s | 19 kB 00:00 2024-09-15T03:57:07.924 INFO:teuthology.orchestra.run.smithi110.stdout:(23/34): python3-jaraco-collections-3.0.0-8.el9 458 kB/s | 23 kB 00:00 2024-09-15T03:57:07.940 INFO:teuthology.orchestra.run.smithi110.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 467 kB/s | 20 kB 00:00 2024-09-15T03:57:07.957 INFO:teuthology.orchestra.run.smithi110.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-15T03:57:07.983 INFO:teuthology.orchestra.run.smithi110.stdout:(26/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-09-15T03:57:08.008 INFO:teuthology.orchestra.run.smithi110.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-09-15T03:57:08.025 INFO:teuthology.orchestra.run.smithi110.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 391 kB/s | 16 kB 00:00 2024-09-15T03:57:08.042 INFO:teuthology.orchestra.run.smithi110.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-15T03:57:08.067 INFO:teuthology.orchestra.run.smithi110.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-15T03:57:08.092 INFO:teuthology.orchestra.run.smithi110.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-09-15T03:57:08.118 INFO:teuthology.orchestra.run.smithi110.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-15T03:57:08.143 INFO:teuthology.orchestra.run.smithi110.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-09-15T03:57:08.168 INFO:teuthology.orchestra.run.smithi110.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-15T03:57:08.172 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:08.172 INFO:teuthology.orchestra.run.smithi110.stdout:Total 13 MB/s | 27 MB 00:02 2024-09-15T03:57:08.301 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:57:08.338 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:57:08.338 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:57:08.598 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:57:08.625 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:57:08.626 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:57:08.626 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T03:57:08.626 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:57:08.626 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:57:08.626 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:08.730 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:57:08.730 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:57:09.007 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:57:09.031 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:57:09.031 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:57:09.031 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T03:57:09.031 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:57:09.032 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:57:09.032 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:09.241 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:57:09.272 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:57:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:57:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T03:57:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:57:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:57:09.272 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:09.555 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:57:10.117 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:57:10.207 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T03:57:10.245 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T03:57:10.278 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T03:57:10.370 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T03:57:10.456 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T03:57:10.497 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T03:57:10.533 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T03:57:10.562 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T03:57:10.597 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T03:57:10.631 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T03:57:10.671 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T03:57:10.718 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T03:57:10.753 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T03:57:10.801 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T03:57:10.834 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:57:10.907 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T03:57:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:57:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T03:57:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T03:57:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T03:57:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T03:57:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T03:57:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T03:57:10.951 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T03:57:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T03:57:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T03:57:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T03:57:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T03:57:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T03:57:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T03:57:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T03:57:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:57:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T03:57:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T03:57:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T03:57:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T03:57:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T03:57:10.952 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T03:57:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T03:57:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T03:57:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T03:57:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T03:57:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T03:57:10.953 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T03:57:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T03:57:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T03:57:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T03:57:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T03:57:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T03:57:10.954 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T03:57:11.052 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T03:57:11.132 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T03:57:11.186 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T03:57:11.233 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T03:57:11.283 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T03:57:11.337 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T03:57:11.532 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T03:57:11.579 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T03:57:11.650 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T03:57:11.737 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T03:57:11.771 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T03:57:11.810 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T03:57:11.810 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:11.810 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:57:11.810 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:11.810 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:11.810 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:11.810 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:11.810 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:11.810 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:11.811 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T03:57:11.811 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T03:57:11.811 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T03:57:11.811 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T03:57:11.811 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T03:57:11.811 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T03:57:11.811 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T03:57:11.811 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T03:57:11.811 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T03:57:11.811 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T03:57:11.811 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T03:57:11.811 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T03:57:11.811 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T03:57:11.812 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T03:57:11.812 INFO:teuthology.orchestra.run.smithi177.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T03:57:11.812 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T03:57:11.812 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T03:57:11.812 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T03:57:11.812 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T03:57:11.812 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T03:57:11.812 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T03:57:11.812 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T03:57:11.813 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T03:57:11.813 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T03:57:11.813 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T03:57:11.813 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T03:57:11.813 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T03:57:11.813 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T03:57:11.813 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:11.813 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:11.893 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T03:57:12.051 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T03:57:12.081 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:57:12.107 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:57:12.107 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:57:12.107 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T03:57:12.108 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:57:12.108 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:57:12.108 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:12.222 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-base 2024-09-15T03:57:12.800 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:54 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:12.889 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:57:12.931 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:12.932 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-15T03:57:12.932 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:13.004 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install cephadm 2024-09-15T03:57:13.416 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:57:13.443 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:57:13.444 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:57:13.444 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T03:57:13.444 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:57:13.444 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:57:13.444 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:13.572 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:55 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:13.702 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:13.702 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:13.703 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-15T03:57:13.703 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:13.703 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:57:13.703 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T03:57:13.703 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:13.703 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:57:13.703 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:13.703 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-15T03:57:13.703 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:13.703 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 83 k 2024-09-15T03:57:13.703 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 375 k 2024-09-15T03:57:13.703 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:57:13.817 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:57:13.841 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:57:13.841 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:57:13.841 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T03:57:13.841 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:57:13.841 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:57:13.841 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:13.898 INFO:teuthology.orchestra.run.smithi177.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 425 kB/s | 83 kB 00:00 2024-09-15T03:57:13.899 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:13.899 INFO:teuthology.orchestra.run.smithi177.stdout:Total 421 kB/s | 83 kB 00:00 2024-09-15T03:57:13.899 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:57:13.904 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:57:13.904 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:57:13.906 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:57:13.906 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:57:13.984 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:57:14.048 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:57:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:57:14.072 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:57:14.072 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T03:57:14.072 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:57:14.072 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:57:14.072 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:14.136 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:57:14.423 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:57:14.528 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:57:14.879 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:57:15.116 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:57:15.116 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:15.116 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:57:15.116 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:15.117 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:15.117 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:15.312 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T03:57:15.886 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:57 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:15.900 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:57:15.900 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T03:57:15.900 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T03:57:15.900 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T03:57:15.900 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T03:57:15.900 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T03:57:15.900 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T03:57:15.900 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T03:57:15.900 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T03:57:15.901 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T03:57:15.901 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T03:57:15.901 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T03:57:15.901 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T03:57:15.901 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T03:57:15.901 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T03:57:15.901 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:57:15.901 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T03:57:15.901 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T03:57:15.901 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T03:57:15.901 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T03:57:15.901 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T03:57:15.902 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T03:57:15.902 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T03:57:15.902 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T03:57:15.902 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T03:57:15.902 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T03:57:15.902 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T03:57:15.902 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T03:57:15.902 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T03:57:15.902 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T03:57:15.902 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T03:57:15.902 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T03:57:15.902 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T03:57:15.902 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T03:57:16.022 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:16.023 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:16.023 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-09-15T03:57:16.023 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:16.023 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:57:16.024 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T03:57:16.024 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:16.024 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:57:16.024 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:16.024 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-15T03:57:16.024 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:16.024 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 137 k 2024-09-15T03:57:16.024 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 379 k 2024-09-15T03:57:16.024 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:57:16.404 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 361 kB/s | 137 kB 00:00 2024-09-15T03:57:16.405 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:16.405 INFO:teuthology.orchestra.run.smithi177.stdout:Total 360 kB/s | 137 kB 00:00 2024-09-15T03:57:16.405 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:57:16.413 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:57:16.413 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:57:16.469 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:57:16.469 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:57:16.614 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:57:16.681 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:57:16.706 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:57:16.706 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:57:16.706 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T03:57:16.706 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:16.768 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T03:57:16.769 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:16.769 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:57:16.769 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:16.769 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:16.769 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:16.769 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:16.769 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:16.769 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:16.769 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T03:57:16.769 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T03:57:16.769 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T03:57:16.769 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T03:57:16.769 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T03:57:16.770 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T03:57:16.770 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T03:57:16.770 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T03:57:16.770 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T03:57:16.770 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T03:57:16.770 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T03:57:16.770 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T03:57:16.770 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T03:57:16.770 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T03:57:16.770 INFO:teuthology.orchestra.run.smithi110.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T03:57:16.770 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T03:57:16.770 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T03:57:16.770 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T03:57:16.771 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T03:57:16.771 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T03:57:16.771 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T03:57:16.771 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T03:57:16.771 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T03:57:16.771 INFO:teuthology.orchestra.run.smithi110.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T03:57:16.771 INFO:teuthology.orchestra.run.smithi110.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T03:57:16.771 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T03:57:16.771 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T03:57:16.771 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T03:57:16.771 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:16.771 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:57:17.151 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base 2024-09-15T03:57:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:57:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:17.648 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:57:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:17.648 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:17.648 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:17.716 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:00 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:57:17.806 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:57:17.848 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:57:17.849 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-15T03:57:17.849 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:57:17.877 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr 2024-09-15T03:57:17.937 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install cephadm 2024-09-15T03:57:18.460 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:00 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:18.499 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:01 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:57:18.553 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:57:18.595 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:18.596 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-15T03:57:18.596 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:18.632 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:57:18.633 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:18.633 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-15T03:57:18.633 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:18.633 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:57:18.633 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T03:57:18.633 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:18.633 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:57:18.633 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:18.633 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-15T03:57:18.634 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:18.634 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 83 k 2024-09-15T03:57:18.634 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 375 k 2024-09-15T03:57:18.634 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:57:18.670 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T03:57:18.998 INFO:teuthology.orchestra.run.smithi110.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 227 kB/s | 83 kB 00:00 2024-09-15T03:57:18.998 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:18.999 INFO:teuthology.orchestra.run.smithi110.stdout:Total 226 kB/s | 83 kB 00:00 2024-09-15T03:57:18.999 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:57:19.004 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:57:19.004 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:57:19.006 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:57:19.006 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:57:19.082 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:57:19.231 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:01 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:19.365 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:19.366 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:19.366 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-15T03:57:19.366 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:19.366 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:57:19.366 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T03:57:19.366 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-15T03:57:19.366 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T03:57:19.366 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T03:57:19.366 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T03:57:19.366 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T03:57:19.367 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T03:57:19.367 INFO:teuthology.orchestra.run.smithi177.stdout:Installing weak dependencies: 2024-09-15T03:57:19.367 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T03:57:19.367 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:19.367 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:57:19.367 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:19.367 INFO:teuthology.orchestra.run.smithi177.stdout:Install 7 Packages 2024-09-15T03:57:19.367 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:19.368 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 3.7 M 2024-09-15T03:57:19.368 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 80 M 2024-09-15T03:57:19.368 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:57:19.557 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:57:19.661 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:57:19.808 INFO:teuthology.orchestra.run.smithi177.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 84 kB/s | 24 kB 00:00 2024-09-15T03:57:19.834 INFO:teuthology.orchestra.run.smithi177.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 46 kB/s | 15 kB 00:00 2024-09-15T03:57:19.876 INFO:teuthology.orchestra.run.smithi177.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-15T03:57:19.901 INFO:teuthology.orchestra.run.smithi177.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-15T03:57:19.926 INFO:teuthology.orchestra.run.smithi177.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-09-15T03:57:19.951 INFO:teuthology.orchestra.run.smithi177.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T03:57:20.000 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:57:20.259 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:57:20.259 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:20.259 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:57:20.259 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:20.259 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:20.259 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:57:20.481 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T03:57:20.535 INFO:teuthology.orchestra.run.smithi177.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 3.3 MB/s | 3.4 MB 00:01 2024-09-15T03:57:20.539 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:20.539 INFO:teuthology.orchestra.run.smithi177.stdout:Total 3.1 MB/s | 3.7 MB 00:01 2024-09-15T03:57:20.539 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:57:20.547 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:57:20.547 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:57:20.629 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:57:20.629 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:57:20.938 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:57:21.011 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T03:57:21.045 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T03:57:21.055 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:04 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:57:21.091 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T03:57:21.123 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T03:57:21.152 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T03:57:21.189 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:57:21.190 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:21.190 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-09-15T03:57:21.190 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:21.190 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:57:21.190 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T03:57:21.190 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:21.191 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:57:21.191 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:21.191 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-15T03:57:21.191 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:21.191 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 137 k 2024-09-15T03:57:21.191 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 379 k 2024-09-15T03:57:21.191 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:57:22.131 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T03:57:22.164 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:57:22.404 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 113 kB/s | 137 kB 00:01 2024-09-15T03:57:22.405 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:22.405 INFO:teuthology.orchestra.run.smithi110.stdout:Total 113 kB/s | 137 kB 00:01 2024-09-15T03:57:22.406 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:57:22.414 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:57:22.414 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:57:22.475 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:57:22.476 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:57:22.630 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:57:22.730 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:57:22.756 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:57:22.756 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:57:22.756 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T03:57:22.756 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:23.177 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:57:23.177 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T03:57:23.178 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T03:57:23.178 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T03:57:23.178 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T03:57:23.178 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T03:57:23.178 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T03:57:23.464 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T03:57:23.518 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:23.519 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:57:23.519 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:23.519 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:23.519 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:23.519 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T03:57:23.519 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T03:57:23.519 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T03:57:23.520 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T03:57:23.520 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:23.520 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:23.682 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:57:23.682 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:23.683 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:57:23.683 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:23.683 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:23.683 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:57:23.707 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T03:57:23.916 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr 2024-09-15T03:57:24.275 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:06 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:24.409 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:24.410 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-09-15T03:57:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-15T03:57:24.410 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-09-15T03:57:24.410 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:57:24.410 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T03:57:24.410 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-15T03:57:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T03:57:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T03:57:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T03:57:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T03:57:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T03:57:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T03:57:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T03:57:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T03:57:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T03:57:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T03:57:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T03:57:24.411 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:24.411 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:57:24.411 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-09-15T03:57:24.412 INFO:teuthology.orchestra.run.smithi177.stdout:Install 12 Packages 2024-09-15T03:57:24.412 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:24.412 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 38 M 2024-09-15T03:57:24.412 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 212 M 2024-09-15T03:57:24.412 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:57:24.489 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:07 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:57:24.581 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:57:24.623 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:57:24.624 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-15T03:57:24.624 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:57:24.699 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T03:57:24.909 INFO:teuthology.orchestra.run.smithi177.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-15T03:57:24.968 INFO:teuthology.orchestra.run.smithi177.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.4 MB/s | 801 kB 00:00 2024-09-15T03:57:24.993 INFO:teuthology.orchestra.run.smithi177.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 395 kB/s | 33 kB 00:00 2024-09-15T03:57:25.035 INFO:teuthology.orchestra.run.smithi177.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-09-15T03:57:25.086 INFO:teuthology.orchestra.run.smithi177.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 768 kB/s | 38 kB 00:00 2024-09-15T03:57:25.261 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:08 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:57:25.396 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:57:25.396 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:25.397 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-15T03:57:25.397 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:25.397 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:57:25.397 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T03:57:25.397 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-15T03:57:25.397 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T03:57:25.397 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T03:57:25.397 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T03:57:25.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T03:57:25.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T03:57:25.398 INFO:teuthology.orchestra.run.smithi110.stdout:Installing weak dependencies: 2024-09-15T03:57:25.398 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T03:57:25.398 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:25.398 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:57:25.398 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:25.398 INFO:teuthology.orchestra.run.smithi110.stdout:Install 7 Packages 2024-09-15T03:57:25.399 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:25.399 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 3.7 M 2024-09-15T03:57:25.399 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 80 M 2024-09-15T03:57:25.400 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:57:25.619 INFO:teuthology.orchestra.run.smithi177.stdout:(6/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 9.3 MB/s | 4.9 MB 00:00 2024-09-15T03:57:25.711 INFO:teuthology.orchestra.run.smithi177.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.0 MB/s | 3.0 MB 00:00 2024-09-15T03:57:25.736 INFO:teuthology.orchestra.run.smithi177.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.0 MB/s | 245 kB 00:00 2024-09-15T03:57:25.820 INFO:teuthology.orchestra.run.smithi177.stdout:(9/12): ceph-mgr-diskprediction-local-17.2.7-16 6.8 MB/s | 7.4 MB 00:01 2024-09-15T03:57:25.845 INFO:teuthology.orchestra.run.smithi177.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.3 MB/s | 250 kB 00:00 2024-09-15T03:57:25.933 INFO:teuthology.orchestra.run.smithi110.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 75 kB/s | 15 kB 00:00 2024-09-15T03:57:25.950 INFO:teuthology.orchestra.run.smithi110.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1643.gd75 116 kB/s | 24 kB 00:00 2024-09-15T03:57:25.983 INFO:teuthology.orchestra.run.smithi110.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 180 kB/s | 9.0 kB 00:00 2024-09-15T03:57:26.000 INFO:teuthology.orchestra.run.smithi110.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 2.0 MB/s | 31 kB 00:00 2024-09-15T03:57:26.017 INFO:teuthology.orchestra.run.smithi110.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 611 kB/s | 41 kB 00:00 2024-09-15T03:57:26.042 INFO:teuthology.orchestra.run.smithi110.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-15T03:57:26.318 INFO:teuthology.orchestra.run.smithi110.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 5.8 MB/s | 3.4 MB 00:00 2024-09-15T03:57:26.321 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:26.321 INFO:teuthology.orchestra.run.smithi110.stdout:Total 4.0 MB/s | 3.7 MB 00:00 2024-09-15T03:57:26.322 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:57:26.330 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:57:26.330 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:57:26.417 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:57:26.418 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:57:26.720 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:57:26.786 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T03:57:26.829 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T03:57:26.884 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T03:57:26.915 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T03:57:26.952 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T03:57:27.029 INFO:teuthology.orchestra.run.smithi177.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.9 MB/s | 5.1 MB 00:01 2024-09-15T03:57:27.454 INFO:teuthology.orchestra.run.smithi177.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-15T03:57:27.455 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:27.455 INFO:teuthology.orchestra.run.smithi177.stdout:Total 13 MB/s | 38 MB 00:03 2024-09-15T03:57:27.812 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:57:27.837 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:57:27.837 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:57:27.953 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T03:57:27.990 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:57:28.355 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:57:28.355 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:57:29.065 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:57:29.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T03:57:29.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T03:57:29.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T03:57:29.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T03:57:29.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T03:57:29.065 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T03:57:29.134 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:57:29.207 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T03:57:29.276 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:57:29.356 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T03:57:29.357 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:29.357 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:57:29.357 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:29.357 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:29.357 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:29.357 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T03:57:29.357 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T03:57:29.357 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T03:57:29.357 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T03:57:29.357 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:29.357 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:57:29.376 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:57:29.411 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T03:57:29.581 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T03:57:29.590 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T03:57:29.627 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T03:57:29.705 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T03:57:30.147 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:13 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:57:30.280 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:57:30.281 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-09-15T03:57:30.281 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-15T03:57:30.281 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-09-15T03:57:30.281 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:57:30.281 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T03:57:30.281 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-15T03:57:30.281 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T03:57:30.282 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T03:57:30.282 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T03:57:30.282 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T03:57:30.282 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T03:57:30.282 INFO:teuthology.orchestra.run.smithi110.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T03:57:30.282 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T03:57:30.282 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T03:57:30.283 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T03:57:30.283 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T03:57:30.283 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T03:57:30.283 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:30.283 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:57:30.283 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================= 2024-09-15T03:57:30.284 INFO:teuthology.orchestra.run.smithi110.stdout:Install 12 Packages 2024-09-15T03:57:30.284 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:30.284 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 38 M 2024-09-15T03:57:30.284 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 212 M 2024-09-15T03:57:30.285 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:57:30.371 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T03:57:30.491 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T03:57:30.893 INFO:teuthology.orchestra.run.smithi110.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 336 kB/s | 191 kB 00:00 2024-09-15T03:57:30.977 INFO:teuthology.orchestra.run.smithi110.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 393 kB/s | 33 kB 00:00 2024-09-15T03:57:31.094 INFO:teuthology.orchestra.run.smithi110.stdout:(3/12): ceph-mgr-diskprediction-local-17.2.7-16 9.6 MB/s | 7.4 MB 00:00 2024-09-15T03:57:31.161 INFO:teuthology.orchestra.run.smithi110.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 274 kB/s | 18 kB 00:00 2024-09-15T03:57:31.244 INFO:teuthology.orchestra.run.smithi110.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-09-15T03:57:31.278 INFO:teuthology.orchestra.run.smithi110.stdout:(6/12): libgfortran-11.5.0-2.el9.x86_64.rpm 840 kB/s | 801 kB 00:00 2024-09-15T03:57:31.303 INFO:teuthology.orchestra.run.smithi110.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 270 kB/s | 38 kB 00:00 2024-09-15T03:57:31.387 INFO:teuthology.orchestra.run.smithi110.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 35 MB/s | 4.9 MB 00:00 2024-09-15T03:57:31.420 INFO:teuthology.orchestra.run.smithi110.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.0 MB/s | 245 kB 00:00 2024-09-15T03:57:31.454 INFO:teuthology.orchestra.run.smithi110.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 3.7 MB/s | 250 kB 00:00 2024-09-15T03:57:31.604 INFO:teuthology.orchestra.run.smithi110.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-09-15T03:57:31.728 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:57:31.796 INFO:teuthology.orchestra.run.smithi110.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-09-15T03:57:31.797 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:31.797 INFO:teuthology.orchestra.run.smithi110.stdout:Total 25 MB/s | 38 MB 00:01 2024-09-15T03:57:32.152 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:57:32.178 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:57:32.178 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:57:32.698 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:57:32.698 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:57:32.783 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T03:57:32.811 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:57:33.454 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:57:33.533 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T03:57:33.582 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:57:33.676 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:57:33.713 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T03:57:33.898 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T03:57:33.936 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T03:57:34.017 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T03:57:34.661 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T03:57:34.782 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T03:57:35.524 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:57:35.524 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T03:57:35.524 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:57:35.525 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:57:35.525 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T03:57:35.525 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T03:57:35.525 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T03:57:35.525 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T03:57:35.525 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T03:57:35.525 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T03:57:35.525 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:57:35.525 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T03:57:35.915 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T03:57:35.915 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:35.915 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:57:35.915 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:35.915 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T03:57:35.916 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T03:57:35.916 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T03:57:35.916 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T03:57:35.916 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T03:57:35.916 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T03:57:35.916 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T03:57:35.916 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T03:57:35.916 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T03:57:35.916 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T03:57:35.916 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T03:57:35.916 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:35.917 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:36.024 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:57:36.302 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-rook 2024-09-15T03:57:36.876 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:18 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:37.011 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:37.012 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:37.012 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-15T03:57:37.012 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:37.012 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:57:37.012 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T03:57:37.013 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-15T03:57:37.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T03:57:37.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T03:57:37.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T03:57:37.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T03:57:37.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T03:57:37.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T03:57:37.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T03:57:37.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T03:57:37.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T03:57:37.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T03:57:37.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T03:57:37.013 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T03:57:37.013 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:37.014 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:57:37.014 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:37.014 INFO:teuthology.orchestra.run.smithi177.stdout:Install 13 Packages 2024-09-15T03:57:37.014 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:37.014 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 2.2 M 2024-09-15T03:57:37.014 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 26 M 2024-09-15T03:57:37.014 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:57:37.061 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T03:57:37.087 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:57:37.270 INFO:teuthology.orchestra.run.smithi177.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 261 kB/s | 19 kB 00:00 2024-09-15T03:57:37.295 INFO:teuthology.orchestra.run.smithi177.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 259 kB/s | 26 kB 00:00 2024-09-15T03:57:37.354 INFO:teuthology.orchestra.run.smithi177.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 222 kB 00:00 2024-09-15T03:57:37.379 INFO:teuthology.orchestra.run.smithi177.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-09-15T03:57:37.404 INFO:teuthology.orchestra.run.smithi177.stdout:(5/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 228 kB/s | 47 kB 00:00 2024-09-15T03:57:37.430 INFO:teuthology.orchestra.run.smithi177.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-15T03:57:37.463 INFO:teuthology.orchestra.run.smithi177.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 279 kB 00:00 2024-09-15T03:57:37.514 INFO:teuthology.orchestra.run.smithi177.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 295 kB/s | 32 kB 00:00 2024-09-15T03:57:37.539 INFO:teuthology.orchestra.run.smithi177.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 130 kB/s | 14 kB 00:00 2024-09-15T03:57:37.564 INFO:teuthology.orchestra.run.smithi177.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-15T03:57:37.589 INFO:teuthology.orchestra.run.smithi177.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-15T03:57:37.615 INFO:teuthology.orchestra.run.smithi177.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-15T03:57:37.648 INFO:teuthology.orchestra.run.smithi177.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.7 MB/s | 1.0 MB 00:00 2024-09-15T03:57:37.651 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:37.651 INFO:teuthology.orchestra.run.smithi177.stdout:Total 3.5 MB/s | 2.2 MB 00:00 2024-09-15T03:57:37.714 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:57:37.729 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:57:37.729 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:57:37.817 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:57:37.817 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:57:38.323 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:57:38.502 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T03:57:38.558 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T03:57:38.622 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T03:57:38.659 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T03:57:38.700 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T03:57:38.802 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T03:57:38.904 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T03:57:38.960 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T03:57:39.681 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T03:57:39.702 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:57:39.702 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T03:57:39.702 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:57:39.702 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:57:39.702 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T03:57:39.702 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T03:57:39.702 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T03:57:39.702 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T03:57:39.703 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T03:57:39.703 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T03:57:39.703 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:57:39.703 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T03:57:39.733 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T03:57:39.766 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T03:57:39.812 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T03:57:39.836 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:57:40.115 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T03:57:40.115 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:40.116 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:57:40.116 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:40.116 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T03:57:40.116 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T03:57:40.116 INFO:teuthology.orchestra.run.smithi110.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T03:57:40.116 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T03:57:40.116 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T03:57:40.116 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T03:57:40.116 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T03:57:40.116 INFO:teuthology.orchestra.run.smithi110.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T03:57:40.116 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T03:57:40.116 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T03:57:40.117 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T03:57:40.117 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:40.117 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:57:40.507 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-rook 2024-09-15T03:57:40.655 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:57:40.655 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T03:57:40.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T03:57:40.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T03:57:40.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T03:57:40.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T03:57:40.656 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T03:57:40.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T03:57:40.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T03:57:40.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T03:57:40.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T03:57:40.657 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T03:57:40.658 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T03:57:40.986 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T03:57:40.987 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:40.987 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:57:40.987 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:40.987 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T03:57:40.987 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T03:57:40.987 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T03:57:40.987 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T03:57:40.987 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T03:57:40.987 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T03:57:40.987 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T03:57:40.987 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T03:57:40.988 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T03:57:40.988 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T03:57:40.988 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T03:57:40.988 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T03:57:40.988 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:40.988 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:41.084 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:24 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:57:41.220 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:57:41.221 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:41.222 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-15T03:57:41.222 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:41.222 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:57:41.222 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T03:57:41.222 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-15T03:57:41.222 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T03:57:41.222 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T03:57:41.222 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T03:57:41.223 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T03:57:41.223 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T03:57:41.223 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T03:57:41.223 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T03:57:41.223 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T03:57:41.223 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T03:57:41.223 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T03:57:41.223 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T03:57:41.223 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T03:57:41.223 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:41.223 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:57:41.223 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:41.223 INFO:teuthology.orchestra.run.smithi110.stdout:Install 13 Packages 2024-09-15T03:57:41.223 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:41.224 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 2.2 M 2024-09-15T03:57:41.224 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 26 M 2024-09-15T03:57:41.224 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:57:41.258 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T03:57:41.622 INFO:teuthology.orchestra.run.smithi110.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 212 kB/s | 19 kB 00:00 2024-09-15T03:57:41.639 INFO:teuthology.orchestra.run.smithi110.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 239 kB/s | 26 kB 00:00 2024-09-15T03:57:41.698 INFO:teuthology.orchestra.run.smithi110.stdout:(3/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 286 kB/s | 47 kB 00:00 2024-09-15T03:57:41.748 INFO:teuthology.orchestra.run.smithi110.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-15T03:57:41.773 INFO:teuthology.orchestra.run.smithi110.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-15T03:57:41.807 INFO:teuthology.orchestra.run.smithi110.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-09-15T03:57:41.825 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:23 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:41.840 INFO:teuthology.orchestra.run.smithi110.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-15T03:57:41.865 INFO:teuthology.orchestra.run.smithi110.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 248 kB/s | 14 kB 00:00 2024-09-15T03:57:41.891 INFO:teuthology.orchestra.run.smithi110.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-15T03:57:41.916 INFO:teuthology.orchestra.run.smithi110.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-15T03:57:41.942 INFO:teuthology.orchestra.run.smithi110.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-15T03:57:41.960 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:41.960 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:41.960 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-15T03:57:41.961 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:41.961 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:57:41.961 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T03:57:41.961 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-15T03:57:41.961 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T03:57:41.961 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T03:57:41.961 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T03:57:41.961 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T03:57:41.961 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T03:57:41.961 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:41.961 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:57:41.961 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:41.961 INFO:teuthology.orchestra.run.smithi177.stdout:Install 6 Packages 2024-09-15T03:57:41.962 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:41.962 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 7.0 M 2024-09-15T03:57:41.962 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 33 M 2024-09-15T03:57:41.962 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:57:41.992 INFO:teuthology.orchestra.run.smithi110.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.1 MB/s | 1.0 MB 00:00 2024-09-15T03:57:42.017 INFO:teuthology.orchestra.run.smithi110.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 893 kB/s | 90 kB 00:00 2024-09-15T03:57:42.019 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:42.020 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.8 MB/s | 2.2 MB 00:00 2024-09-15T03:57:42.082 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:57:42.097 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:57:42.097 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:57:42.187 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:57:42.188 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:57:42.516 INFO:teuthology.orchestra.run.smithi177.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 612 kB/s | 124 kB 00:00 2024-09-15T03:57:42.584 INFO:teuthology.orchestra.run.smithi177.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.0 MB/s | 548 kB 00:00 2024-09-15T03:57:42.609 INFO:teuthology.orchestra.run.smithi177.stdout:(3/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 842 kB/s | 248 kB 00:00 2024-09-15T03:57:42.626 INFO:teuthology.orchestra.run.smithi177.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.3 MB/s | 58 kB 00:00 2024-09-15T03:57:42.668 INFO:teuthology.orchestra.run.smithi177.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.2 MB/s | 74 kB 00:00 2024-09-15T03:57:42.704 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:57:42.872 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T03:57:42.927 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T03:57:42.982 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T03:57:43.028 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T03:57:43.070 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T03:57:43.171 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T03:57:43.290 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T03:57:43.346 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T03:57:44.066 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T03:57:44.111 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T03:57:44.145 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T03:57:44.190 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T03:57:44.214 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:57:44.818 INFO:teuthology.orchestra.run.smithi177.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.4 MB/s | 6.0 MB 00:02 2024-09-15T03:57:44.819 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:44.820 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.4 MB/s | 7.0 MB 00:02 2024-09-15T03:57:44.891 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:57:44.903 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:57:44.903 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:57:44.917 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:57:44.917 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T03:57:44.917 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T03:57:44.917 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T03:57:44.917 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T03:57:44.917 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T03:57:44.917 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T03:57:44.917 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T03:57:44.917 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T03:57:44.918 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T03:57:44.918 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T03:57:44.918 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T03:57:44.918 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T03:57:45.002 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:57:45.002 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:57:45.247 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T03:57:45.247 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:45.247 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:57:45.247 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:45.247 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T03:57:45.247 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T03:57:45.247 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T03:57:45.248 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T03:57:45.248 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T03:57:45.248 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T03:57:45.248 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T03:57:45.248 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T03:57:45.248 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T03:57:45.248 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T03:57:45.248 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T03:57:45.248 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T03:57:45.248 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:45.248 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:57:45.309 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:57:45.474 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T03:57:45.526 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T03:57:45.527 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T03:57:46.050 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T03:57:46.084 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:29 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:57:46.139 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T03:57:46.194 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T03:57:46.217 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:57:46.217 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:46.218 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-15T03:57:46.218 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:46.218 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:57:46.218 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T03:57:46.218 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-15T03:57:46.218 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T03:57:46.218 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T03:57:46.218 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T03:57:46.218 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T03:57:46.218 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T03:57:46.218 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:46.218 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:57:46.219 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:46.219 INFO:teuthology.orchestra.run.smithi110.stdout:Install 6 Packages 2024-09-15T03:57:46.219 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:46.219 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 7.0 M 2024-09-15T03:57:46.219 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 33 M 2024-09-15T03:57:46.219 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:57:46.223 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:57:46.696 INFO:teuthology.orchestra.run.smithi110.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-09-15T03:57:46.721 INFO:teuthology.orchestra.run.smithi110.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 605 kB/s | 124 kB 00:00 2024-09-15T03:57:46.763 INFO:teuthology.orchestra.run.smithi110.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.4 MB/s | 58 kB 00:00 2024-09-15T03:57:46.788 INFO:teuthology.orchestra.run.smithi110.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.1 MB/s | 74 kB 00:00 2024-09-15T03:57:46.822 INFO:teuthology.orchestra.run.smithi110.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.3 MB/s | 548 kB 00:00 2024-09-15T03:57:46.922 INFO:teuthology.orchestra.run.smithi110.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-15T03:57:46.925 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:46.925 INFO:teuthology.orchestra.run.smithi110.stdout:Total 9.9 MB/s | 7.0 MB 00:00 2024-09-15T03:57:46.928 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:57:46.928 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T03:57:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T03:57:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T03:57:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T03:57:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T03:57:46.995 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:57:47.007 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:57:47.007 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:57:47.105 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:57:47.106 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:57:47.194 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T03:57:47.194 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:47.194 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:57:47.195 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:47.195 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T03:57:47.195 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T03:57:47.195 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T03:57:47.195 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T03:57:47.195 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T03:57:47.195 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:47.195 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:47.416 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:57:47.458 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-fuse 2024-09-15T03:57:47.588 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T03:57:47.649 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T03:57:48.033 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:30 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:48.167 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:48.167 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-15T03:57:48.167 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:48.167 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:57:48.167 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T03:57:48.167 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-09-15T03:57:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T03:57:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:48.168 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:57:48.168 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:48.168 INFO:teuthology.orchestra.run.smithi177.stdout:Install 2 Packages 2024-09-15T03:57:48.168 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:48.169 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 893 k 2024-09-15T03:57:48.169 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.6 M 2024-09-15T03:57:48.169 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:57:48.183 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T03:57:48.269 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T03:57:48.325 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T03:57:48.335 INFO:teuthology.orchestra.run.smithi177.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 668 kB/s | 80 kB 00:00 2024-09-15T03:57:48.361 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:57:48.485 INFO:teuthology.orchestra.run.smithi177.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.9 MB/s | 813 kB 00:00 2024-09-15T03:57:48.486 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:48.486 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.7 MB/s | 893 kB 00:00 2024-09-15T03:57:48.498 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:57:48.519 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:57:48.519 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:57:48.598 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:57:48.598 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:57:48.784 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:57:48.930 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T03:57:49.008 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:57:49.082 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:57:49.082 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T03:57:49.082 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T03:57:49.082 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T03:57:49.082 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T03:57:49.082 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T03:57:49.353 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T03:57:49.353 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:49.353 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:57:49.353 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:49.353 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T03:57:49.353 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T03:57:49.353 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T03:57:49.353 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T03:57:49.354 INFO:teuthology.orchestra.run.smithi110.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T03:57:49.354 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:49.354 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:57:49.628 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:57:49.629 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T03:57:49.638 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-fuse 2024-09-15T03:57:49.886 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T03:57:49.886 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:49.886 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:57:49.886 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T03:57:49.886 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:49.886 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:50.189 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-volume 2024-09-15T03:57:50.201 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:33 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:57:50.334 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:57:50.334 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:50.334 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-15T03:57:50.334 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:50.334 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:57:50.335 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T03:57:50.335 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-09-15T03:57:50.335 INFO:teuthology.orchestra.run.smithi110.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T03:57:50.335 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:50.335 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:57:50.335 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:50.335 INFO:teuthology.orchestra.run.smithi110.stdout:Install 2 Packages 2024-09-15T03:57:50.335 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:50.336 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 893 k 2024-09-15T03:57:50.336 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.6 M 2024-09-15T03:57:50.336 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:57:50.458 INFO:teuthology.orchestra.run.smithi110.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 784 kB/s | 80 kB 00:00 2024-09-15T03:57:50.634 INFO:teuthology.orchestra.run.smithi110.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.9 MB/s | 813 kB 00:00 2024-09-15T03:57:50.634 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:50.635 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.9 MB/s | 893 kB 00:00 2024-09-15T03:57:50.647 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:57:50.667 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:57:50.668 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:57:50.747 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:57:50.748 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:57:50.770 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:32 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:50.906 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:50.907 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:50.907 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-09-15T03:57:50.907 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:50.907 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:57:50.907 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T03:57:50.907 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:50.907 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:57:50.907 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:50.908 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-15T03:57:50.908 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:50.908 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 270 k 2024-09-15T03:57:50.908 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 1.3 M 2024-09-15T03:57:50.908 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:57:50.934 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:57:51.080 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T03:57:51.140 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 1.1 MB/s | 270 kB 00:00 2024-09-15T03:57:51.141 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:51.141 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-15T03:57:51.141 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:57:51.149 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:57:51.149 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:57:51.150 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:57:51.176 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:57:51.177 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:57:51.385 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:57:51.450 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:57:51.466 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:57:51.466 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:57:51.466 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T03:57:51.466 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:51.861 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:57:51.861 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T03:57:52.157 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T03:57:52.157 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:52.157 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:57:52.157 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T03:57:52.157 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:52.157 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:57:52.415 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-volume 2024-09-15T03:57:52.500 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:57:52.500 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:52.500 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:57:52.500 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:52.500 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:52.500 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:52.794 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados-devel 2024-09-15T03:57:52.986 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:57:53.122 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:57:53.122 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:53.122 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-09-15T03:57:53.123 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:53.123 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:57:53.123 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T03:57:53.123 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:53.123 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:57:53.123 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:53.123 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-15T03:57:53.123 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:53.123 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 270 k 2024-09-15T03:57:53.123 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.3 M 2024-09-15T03:57:53.123 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:57:53.379 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:35 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:53.513 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:53.514 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-09-15T03:57:53.514 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:53.514 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:57:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T03:57:53.514 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:53.515 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:57:53.515 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:53.515 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-15T03:57:53.515 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:53.515 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 126 k 2024-09-15T03:57:53.515 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 448 k 2024-09-15T03:57:53.515 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:57:53.673 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 492 kB/s | 270 kB 00:00 2024-09-15T03:57:53.673 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:53.674 INFO:teuthology.orchestra.run.smithi110.stdout:Total 490 kB/s | 270 kB 00:00 2024-09-15T03:57:53.674 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:57:53.682 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:57:53.682 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:57:53.716 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:57:53.717 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:57:53.927 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:57:53.969 INFO:teuthology.orchestra.run.smithi177.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 278 kB/s | 126 kB 00:00 2024-09-15T03:57:53.970 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:53.970 INFO:teuthology.orchestra.run.smithi177.stdout:Total 277 kB/s | 126 kB 00:00 2024-09-15T03:57:53.970 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:57:53.975 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:57:53.976 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:57:53.983 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:57:53.999 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:57:53.999 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:57:53.999 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T03:57:53.999 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:54.039 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:57:54.040 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:57:54.191 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:57:54.292 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:57:54.732 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:57:54.997 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:57:54.997 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:54.997 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:57:54.997 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:54.997 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:54.997 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:55.036 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:57:55.036 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:55.036 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:57:55.037 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:57:55.037 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:55.037 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:57:55.253 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs2 2024-09-15T03:57:55.329 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados-devel 2024-09-15T03:57:55.833 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:55.903 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:57:55.925 INFO:teuthology.orchestra.run.smithi177.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:57:55.969 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:55.970 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-15T03:57:55.970 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:56.035 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs-devel 2024-09-15T03:57:56.038 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:57:56.038 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:56.038 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-09-15T03:57:56.038 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:56.038 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:57:56.038 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T03:57:56.039 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:56.039 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:57:56.039 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:56.039 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-15T03:57:56.039 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:56.039 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 126 k 2024-09-15T03:57:56.039 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 448 k 2024-09-15T03:57:56.039 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:57:56.587 INFO:teuthology.orchestra.run.smithi110.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 230 kB/s | 126 kB 00:00 2024-09-15T03:57:56.588 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:56.588 INFO:teuthology.orchestra.run.smithi110.stdout:Total 230 kB/s | 126 kB 00:00 2024-09-15T03:57:56.588 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:57:56.594 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:57:56.594 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:57:56.600 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:56.659 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:57:56.659 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:57:56.732 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:56.733 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:56.733 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-09-15T03:57:56.733 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:56.733 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:57:56.733 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:57:56.733 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:56.734 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:57:56.734 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:57:56.734 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-15T03:57:56.734 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:56.734 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 25 k 2024-09-15T03:57:56.734 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 110 k 2024-09-15T03:57:56.734 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:57:56.809 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:57:56.916 INFO:teuthology.orchestra.run.smithi177.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 139 kB/s | 25 kB 00:00 2024-09-15T03:57:56.916 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:56.917 INFO:teuthology.orchestra.run.smithi177.stdout:Total 138 kB/s | 25 kB 00:00 2024-09-15T03:57:56.917 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:57:56.917 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:57:56.918 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:57:56.918 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:57:56.920 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:57:56.920 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:57:57.004 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:57:57.103 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:57:57.318 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:57:57.497 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:57:57.588 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:57:57.589 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:57.589 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:57:57.589 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:57.589 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:57.589 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:57:57.734 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:57:57.735 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:57.735 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:57:57.735 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:57:57.735 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:57:57.735 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:57.842 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs2 2024-09-15T03:57:57.915 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados2 2024-09-15T03:57:58.420 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:41 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:57:58.484 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:58.510 INFO:teuthology.orchestra.run.smithi110.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:57:58.553 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:57:58.554 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-15T03:57:58.554 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:57:58.573 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:57:58.616 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:58.617 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-15T03:57:58.617 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:58.629 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs-devel 2024-09-15T03:57:58.694 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librbd1 2024-09-15T03:57:59.193 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:42 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:57:59.261 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:41 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:57:59.325 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:57:59.325 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:59.325 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-09-15T03:57:59.326 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:59.326 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:57:59.326 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:57:59.326 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:59.326 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:57:59.326 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:57:59.326 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-15T03:57:59.326 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:57:59.327 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 25 k 2024-09-15T03:57:59.327 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 110 k 2024-09-15T03:57:59.327 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:57:59.352 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:57:59.394 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:57:59.396 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-15T03:57:59.396 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:57:59.467 INFO:teuthology.orchestra.run.smithi110.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 178 kB/s | 25 kB 00:00 2024-09-15T03:57:59.467 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:57:59.468 INFO:teuthology.orchestra.run.smithi110.stdout:Total 176 kB/s | 25 kB 00:00 2024-09-15T03:57:59.468 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:57:59.469 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rados 2024-09-15T03:57:59.472 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:57:59.472 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:57:59.473 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:57:59.473 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:57:59.554 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:57:59.680 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:00.048 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:42 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:58:00.079 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:00.138 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:58:00.180 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:58:00.181 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-15T03:58:00.182 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:58:00.253 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rgw 2024-09-15T03:58:00.310 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:00.310 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:00.310 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:58:00.310 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:58:00.310 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:00.310 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:58:00.505 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados2 2024-09-15T03:58:00.820 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:42 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:58:00.910 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:58:00.953 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:58:00.954 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-15T03:58:00.954 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:58:01.036 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-cephfs 2024-09-15T03:58:01.072 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:44 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:58:01.160 INFO:teuthology.orchestra.run.smithi110.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:58:01.202 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:58:01.203 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-15T03:58:01.203 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:58:01.288 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librbd1 2024-09-15T03:58:01.618 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:43 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:58:01.718 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:58:01.761 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:58:01.762 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-15T03:58:01.762 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:58:01.836 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rbd 2024-09-15T03:58:01.850 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:44 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:58:01.942 INFO:teuthology.orchestra.run.smithi110.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:58:01.985 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:58:01.986 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-15T03:58:01.986 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:58:02.072 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rados 2024-09-15T03:58:02.398 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:44 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:58:02.488 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:58:02.530 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:58:02.531 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-09-15T03:58:02.531 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:58:02.599 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-fuse 2024-09-15T03:58:02.631 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:45 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:58:02.721 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:58:02.764 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:58:02.765 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-15T03:58:02.765 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:58:02.847 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rgw 2024-09-15T03:58:03.175 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:45 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:58:03.318 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:58:03.318 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:58:03.318 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-09-15T03:58:03.318 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:58:03.319 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:58:03.319 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T03:58:03.319 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:03.319 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:58:03.319 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:58:03.319 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-15T03:58:03.319 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:03.319 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 82 k 2024-09-15T03:58:03.319 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 222 k 2024-09-15T03:58:03.319 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:58:03.406 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:46 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:58:03.496 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:58:03.539 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:58:03.540 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-15T03:58:03.540 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:58:03.630 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-cephfs 2024-09-15T03:58:03.709 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 210 kB/s | 82 kB 00:00 2024-09-15T03:58:03.710 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:58:03.710 INFO:teuthology.orchestra.run.smithi177.stdout:Total 209 kB/s | 82 kB 00:00 2024-09-15T03:58:03.710 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:58:03.718 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:58:03.718 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:58:03.783 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:58:03.783 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:58:03.930 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:58:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:04.193 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:47 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:58:04.283 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:58:04.325 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:58:04.327 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-15T03:58:04.327 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:58:04.421 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rbd 2024-09-15T03:58:04.435 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:04.689 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:04.689 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:04.689 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:58:04.689 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:58:04.689 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:04.689 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:58:04.978 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-mirror 2024-09-15T03:58:04.983 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:47 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:58:05.073 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:58:05.115 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:58:05.116 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2024-09-15T03:58:05.116 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:58:05.210 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-fuse 2024-09-15T03:58:05.559 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:47 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:58:05.699 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:58:05.699 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:58:05.700 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-09-15T03:58:05.700 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:58:05.700 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:58:05.700 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:58:05.700 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:05.700 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:58:05.700 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:58:05.701 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-15T03:58:05.701 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:05.701 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 2.9 M 2024-09-15T03:58:05.701 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 12 M 2024-09-15T03:58:05.701 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:58:05.776 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:48 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:58:05.908 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:58:05.909 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:58:05.909 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-09-15T03:58:05.909 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:58:05.909 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:58:05.909 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T03:58:05.909 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:05.910 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:58:05.910 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:58:05.910 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-15T03:58:05.910 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:05.910 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 82 k 2024-09-15T03:58:05.910 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 222 k 2024-09-15T03:58:05.910 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:58:06.187 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 295 kB/s | 82 kB 00:00 2024-09-15T03:58:06.188 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:58:06.188 INFO:teuthology.orchestra.run.smithi110.stdout:Total 293 kB/s | 82 kB 00:00 2024-09-15T03:58:06.188 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:58:06.195 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:58:06.195 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:58:06.258 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:58:06.259 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:58:06.402 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:58:06.517 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:06.579 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 3.3 MB/s | 2.9 MB 00:00 2024-09-15T03:58:06.580 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:58:06.580 INFO:teuthology.orchestra.run.smithi177.stdout:Total 3.3 MB/s | 2.9 MB 00:00 2024-09-15T03:58:06.580 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:58:06.590 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:58:06.590 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:58:06.671 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:58:06.671 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:58:06.883 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:07.085 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:58:07.146 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:07.147 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:07.147 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:07.147 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:58:07.147 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:58:07.147 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:07.147 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:58:07.172 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:07.172 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:58:07.172 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T03:58:07.173 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:58:07.173 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:58:07.173 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:07.465 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-mirror 2024-09-15T03:58:08.039 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:51 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:58:08.181 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:58:08.181 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:58:08.181 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2024-09-15T03:58:08.182 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:58:08.182 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:58:08.182 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:58:08.182 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:08.182 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:58:08.182 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:58:08.182 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-15T03:58:08.182 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:08.182 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 2.9 M 2024-09-15T03:58:08.182 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 12 M 2024-09-15T03:58:08.182 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:58:08.271 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:08.271 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:08.272 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:58:08.272 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:58:08.272 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:08.272 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:58:08.501 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 9.2 MB/s | 2.9 MB 00:00 2024-09-15T03:58:08.502 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:58:08.502 INFO:teuthology.orchestra.run.smithi110.stdout:Total 9.2 MB/s | 2.9 MB 00:00 2024-09-15T03:58:08.502 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:58:08.510 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-nbd 2024-09-15T03:58:08.511 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:58:08.511 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:58:08.594 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:58:08.594 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:58:09.027 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:58:09.077 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:09.097 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:51 ago on Sun 15 Sep 2024 03:56:18 AM UTC. 2024-09-15T03:58:09.104 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:09.104 INFO:teuthology.orchestra.run.smithi110.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:58:09.104 INFO:teuthology.orchestra.run.smithi110.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T03:58:09.104 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:58:09.104 INFO:teuthology.orchestra.run.smithi110.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:58:09.104 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:09.231 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-09-15T03:58:09.231 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:58:09.231 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-09-15T03:58:09.231 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:58:09.231 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-09-15T03:58:09.232 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T03:58:09.232 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:09.232 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-09-15T03:58:09.232 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-09-15T03:58:09.232 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-09-15T03:58:09.232 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:09.232 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 165 k 2024-09-15T03:58:09.233 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 477 k 2024-09-15T03:58:09.233 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-09-15T03:58:09.440 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 796 kB/s | 165 kB 00:00 2024-09-15T03:58:09.441 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:58:09.441 INFO:teuthology.orchestra.run.smithi177.stdout:Total 790 kB/s | 165 kB 00:00 2024-09-15T03:58:09.441 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-09-15T03:58:09.449 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-09-15T03:58:09.449 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-09-15T03:58:09.512 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-09-15T03:58:09.513 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-09-15T03:58:09.669 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-09-15T03:58:09.779 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:10.210 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:10.210 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:10.210 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:58:10.210 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:58:10.210 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:10.210 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:58:10.212 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:10.462 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-nbd 2024-09-15T03:58:10.482 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:10.482 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:10.483 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-09-15T03:58:10.483 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:58:10.483 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:10.483 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-09-15T03:58:10.727 DEBUG:teuthology.parallel:result is None 2024-09-15T03:58:11.043 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:54 ago on Sun 15 Sep 2024 03:56:17 AM UTC. 2024-09-15T03:58:11.179 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-09-15T03:58:11.180 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:58:11.180 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-09-15T03:58:11.180 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:58:11.180 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-09-15T03:58:11.180 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T03:58:11.180 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:11.181 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-09-15T03:58:11.181 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-09-15T03:58:11.181 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-09-15T03:58:11.181 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:11.181 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 165 k 2024-09-15T03:58:11.181 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 477 k 2024-09-15T03:58:11.181 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-09-15T03:58:11.397 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 766 kB/s | 165 kB 00:00 2024-09-15T03:58:11.397 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:58:11.398 INFO:teuthology.orchestra.run.smithi110.stdout:Total 761 kB/s | 165 kB 00:00 2024-09-15T03:58:11.398 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-09-15T03:58:11.405 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-09-15T03:58:11.405 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-09-15T03:58:11.468 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-09-15T03:58:11.469 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-09-15T03:58:11.627 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-09-15T03:58:11.726 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:12.153 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:12.442 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:58:12.442 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:12.442 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-09-15T03:58:12.443 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:58:12.443 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:12.443 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-09-15T03:58:12.690 DEBUG:teuthology.parallel:result is None 2024-09-15T03:58:12.691 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:58:12.838 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T03:58:12.859 INFO:teuthology.orchestra.run.smithi110.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T03:58:12.860 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T03:58:12.860 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T03:58:12.862 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:58:13.001 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T03:58:13.027 INFO:teuthology.orchestra.run.smithi177.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T03:58:13.027 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T03:58:13.027 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T03:58:13.029 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T03:58:13.029 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:13.029 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:58:13.056 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:13.056 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:58:13.097 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T03:58:13.097 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:13.098 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:58:13.122 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:58:13.188 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:13.188 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:58:13.218 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:58:13.286 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T03:58:13.287 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:13.287 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:58:13.312 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:58:13.378 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:13.378 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:58:13.406 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:58:13.473 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T03:58:13.474 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:13.474 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:58:13.498 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:58:13.563 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:13.563 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:58:13.591 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:58:13.658 INFO:teuthology.run_tasks:Running task ceph... 2024-09-15T03:58:13.733 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-15T03:58:13.733 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2024-09-15T03:58:13.735 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /var/log/ceph 2024-09-15T03:58:13.760 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-15T03:58:13.761 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T03:58:13.801 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T03:58:13.826 INFO:tasks.ceph:Creating extra log directories... 2024-09-15T03:58:13.826 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T03:58:13.866 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T03:58:13.900 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-15T03:58:13.900 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd sloppy crc': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-15T03:58:13.900 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905333', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_latest} thrashers/sync workloads/pool-create-delete}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7905333', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 2126, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi110.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF+lXPJo6D5KxJc4ZEcTFLJnShYZGsoAQZ5560lCP61/8HPSEc+qrfLMHp6ht69e3JP4vEdqnNNEZ7UAArHI4eY=', 'smithi177.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFvpMn36bc2JmYB6aiw9Btz1iUb5Y59dE12hcrW26b6QKeWr74W6Hsg5dMUJU8lbRAdU+RTOLVOc6+j7j9pKW/o='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_00:54:39', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-15T03:58:13.901 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T03:58:13.931 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T03:58:13.948 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T03:58:13.986 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T03:58:14.013 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:14.013 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T03:58:14.064 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T03:58:14.064 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2024-09-15T03:58:14.119 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T03:58:14.119 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:58:14.120 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-15T03:58:14.120 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:58:14.120 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:58:14.120 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-09-15 03:58:09.367607845 +0000 2024-09-15T03:58:14.120 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-09-15 03:55:38.757858999 +0000 2024-09-15T03:58:14.120 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-09-15 03:55:38.757858999 +0000 2024-09-15T03:58:14.120 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-09-15 03:55:38.757858999 +0000 2024-09-15T03:58:14.120 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T03:58:14.183 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-09-15T03:58:14.183 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-09-15T03:58:14.183 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000102221 s, 5.0 MB/s 2024-09-15T03:58:14.184 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T03:58:14.241 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2024-09-15T03:58:14.297 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T03:58:14.297 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:58:14.297 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-15T03:58:14.297 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:58:14.297 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:58:14.298 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-09-15 03:58:09.367607845 +0000 2024-09-15T03:58:14.298 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-09-15 03:55:39.070853956 +0000 2024-09-15T03:58:14.298 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-09-15 03:55:39.070853956 +0000 2024-09-15T03:58:14.298 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-09-15 03:55:39.070853956 +0000 2024-09-15T03:58:14.298 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T03:58:14.361 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-09-15T03:58:14.361 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-09-15T03:58:14.361 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000100061 s, 5.1 MB/s 2024-09-15T03:58:14.362 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T03:58:14.419 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2024-09-15T03:58:14.474 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T03:58:14.474 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:58:14.474 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-15T03:58:14.474 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:58:14.474 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:58:14.474 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-09-15 03:58:09.368607828 +0000 2024-09-15T03:58:14.474 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-09-15 03:55:39.377849010 +0000 2024-09-15T03:58:14.474 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-09-15 03:55:39.377849010 +0000 2024-09-15T03:58:14.474 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-09-15 03:55:39.377849010 +0000 2024-09-15T03:58:14.475 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T03:58:14.537 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-09-15T03:58:14.537 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-09-15T03:58:14.537 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 8.102e-05 s, 6.3 MB/s 2024-09-15T03:58:14.538 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T03:58:14.594 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2024-09-15T03:58:14.650 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T03:58:14.650 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:58:14.650 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-15T03:58:14.650 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:58:14.651 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:58:14.651 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-09-15 03:58:09.368607828 +0000 2024-09-15T03:58:14.651 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-09-15 03:55:39.702843774 +0000 2024-09-15T03:58:14.651 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-09-15 03:55:39.702843774 +0000 2024-09-15T03:58:14.651 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-09-15 03:55:39.702843774 +0000 2024-09-15T03:58:14.651 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T03:58:14.714 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-09-15T03:58:14.714 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-09-15T03:58:14.714 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000118577 s, 4.3 MB/s 2024-09-15T03:58:14.715 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T03:58:14.771 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T03:58:14.772 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:14.772 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T03:58:14.789 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T03:58:14.790 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-09-15T03:58:14.847 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T03:58:14.847 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:58:14.847 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-15T03:58:14.847 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:58:14.847 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:58:14.847 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-15 03:58:07.409561583 +0000 2024-09-15T03:58:14.847 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-15 03:55:39.227460122 +0000 2024-09-15T03:58:14.847 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-15 03:55:39.227460122 +0000 2024-09-15T03:58:14.848 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-09-15 03:55:39.227460122 +0000 2024-09-15T03:58:14.848 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T03:58:14.912 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-15T03:58:14.913 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-15T03:58:14.913 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000269865 s, 1.9 MB/s 2024-09-15T03:58:14.914 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T03:58:14.971 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-09-15T03:58:15.027 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T03:58:15.028 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:58:15.028 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-15T03:58:15.028 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:58:15.028 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:58:15.028 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-15 03:58:07.409561583 +0000 2024-09-15T03:58:15.028 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-15 03:55:39.554451079 +0000 2024-09-15T03:58:15.028 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-15 03:55:39.554451079 +0000 2024-09-15T03:58:15.028 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-09-15 03:55:39.554451079 +0000 2024-09-15T03:58:15.029 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T03:58:15.092 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-15T03:58:15.092 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-15T03:58:15.092 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000227043 s, 2.3 MB/s 2024-09-15T03:58:15.093 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T03:58:15.150 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-09-15T03:58:15.206 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T03:58:15.206 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:58:15.206 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-15T03:58:15.206 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:58:15.206 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:58:15.206 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-15 03:58:07.409561583 +0000 2024-09-15T03:58:15.207 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-15 03:55:39.867442423 +0000 2024-09-15T03:58:15.207 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-15 03:55:39.867442423 +0000 2024-09-15T03:58:15.207 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-09-15 03:55:39.867442423 +0000 2024-09-15T03:58:15.207 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T03:58:15.270 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-15T03:58:15.270 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-15T03:58:15.270 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.00017036 s, 3.0 MB/s 2024-09-15T03:58:15.271 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T03:58:15.327 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-09-15T03:58:15.382 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T03:58:15.382 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:58:15.382 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-15T03:58:15.382 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:58:15.382 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T03:58:15.382 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-15 03:58:07.410561556 +0000 2024-09-15T03:58:15.383 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-15 03:55:40.195433352 +0000 2024-09-15T03:58:15.383 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-15 03:55:40.195433352 +0000 2024-09-15T03:58:15.383 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-09-15 03:55:40.195433352 +0000 2024-09-15T03:58:15.383 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T03:58:15.445 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-15T03:58:15.445 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-15T03:58:15.445 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000190268 s, 2.7 MB/s 2024-09-15T03:58:15.446 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T03:58:15.503 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T03:58:15.503 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-15T03:58:15.503 INFO:tasks.ceph:Generating config... 2024-09-15T03:58:15.505 INFO:tasks.ceph:[client] debug monc = 20 2024-09-15T03:58:15.505 INFO:tasks.ceph:[client] debug ms = 1 2024-09-15T03:58:15.505 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-15T03:58:15.505 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-15T03:58:15.505 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-15T03:58:15.505 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-15T03:58:15.505 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-15T03:58:15.505 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-15T03:58:15.505 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-15T03:58:15.506 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-15T03:58:15.506 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-15T03:58:15.506 INFO:tasks.ceph:[global] ms type = async 2024-09-15T03:58:15.506 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-15T03:58:15.506 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-15T03:58:15.506 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-15T03:58:15.506 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-15T03:58:15.506 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-15T03:58:15.506 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-15T03:58:15.506 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-15T03:58:15.506 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-15T03:58:15.506 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-15T03:58:15.506 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-15T03:58:15.506 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-15T03:58:15.507 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-15T03:58:15.507 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-15T03:58:15.507 INFO:tasks.ceph:[mon] paxos min = 10 2024-09-15T03:58:15.507 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-15T03:58:15.507 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-09-15T03:58:15.507 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-15T03:58:15.507 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-15T03:58:15.507 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-15T03:58:15.507 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-15T03:58:15.507 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-15T03:58:15.507 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-15T03:58:15.507 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-15T03:58:15.507 INFO:tasks.ceph:[osd] osd objectstore = filestore 2024-09-15T03:58:15.507 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-15T03:58:15.508 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-15T03:58:15.508 INFO:tasks.ceph:[osd] osd sloppy crc = True 2024-09-15T03:58:15.508 INFO:tasks.ceph:Setting up mon.a... 2024-09-15T03:58:15.508 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-15T03:58:15.576 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.keyring 2024-09-15T03:58:15.578 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-15T03:58:15.620 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:58:15.686 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.110'), ('mon.c', '[v2:172.21.15.110:3301,v1:172.21.15.110:6790]'), ('mon.b', '172.21.15.177')] 2024-09-15T03:58:15.687 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790],172.21.15.177', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd sloppy crc': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-15T03:58:15.687 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:15.688 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T03:58:15.743 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.110 --addv c '[v2:172.21.15.110:3301,v1:172.21.15.110:6790]' --add b 172.21.15.177 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:58:15.825 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:58:15.825 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: generated fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:15.825 INFO:teuthology.orchestra.run.smithi110.stdout:setting min_mon_release = octopus 2024-09-15T03:58:15.825 INFO:teuthology.orchestra.run.smithi110.stdout:epoch 0 2024-09-15T03:58:15.825 INFO:teuthology.orchestra.run.smithi110.stdout:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:15.826 INFO:teuthology.orchestra.run.smithi110.stdout:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:15.826 INFO:teuthology.orchestra.run.smithi110.stdout:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:15.826 INFO:teuthology.orchestra.run.smithi110.stdout:min_mon_release 15 (octopus) 2024-09-15T03:58:15.826 INFO:teuthology.orchestra.run.smithi110.stdout:election_strategy: 1 2024-09-15T03:58:15.826 INFO:teuthology.orchestra.run.smithi110.stdout:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:15.826 INFO:teuthology.orchestra.run.smithi110.stdout:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:15.826 INFO:teuthology.orchestra.run.smithi110.stdout:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:15.826 INFO:teuthology.orchestra.run.smithi110.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-15T03:58:15.827 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T03:58:15.883 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 802f2aac-d77f-438d-80e9-5e0c23519086... 2024-09-15T03:58:15.884 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T03:58:15.926 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T03:58:15.967 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2024-09-15T03:58:15.967 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2024-09-15T03:58:15.967 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T03:58:15.967 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2024-09-15T03:58:15.967 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.967 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2024-09-15T03:58:15.967 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.967 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2024-09-15T03:58:15.967 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.967 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2024-09-15T03:58:15.967 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2024-09-15T03:58:15.968 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.968 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2024-09-15T03:58:15.968 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2024-09-15T03:58:15.968 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.968 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T03:58:15.968 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2024-09-15T03:58:15.968 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.968 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T03:58:15.968 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T03:58:15.968 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.968 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2024-09-15T03:58:15.968 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.968 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2024-09-15T03:58:15.969 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2024-09-15T03:58:15.969 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2024-09-15T03:58:15.969 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2024-09-15T03:58:15.969 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2024-09-15T03:58:15.969 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2024-09-15T03:58:15.969 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2024-09-15T03:58:15.969 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T03:58:15.969 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T03:58:15.969 INFO:teuthology.orchestra.run.smithi110.stdout: mon_allow_pool_size_one = true 2024-09-15T03:58:15.969 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.969 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T03:58:15.969 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.969 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2024-09-15T03:58:15.970 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.970 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2024-09-15T03:58:15.970 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.970 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2024-09-15T03:58:15.970 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2024-09-15T03:58:15.970 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2024-09-15T03:58:15.970 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = "172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790],172.21.15.177" 2024-09-15T03:58:15.970 INFO:teuthology.orchestra.run.smithi110.stdout: mon client directed command retry = 5 2024-09-15T03:58:15.970 INFO:teuthology.orchestra.run.smithi110.stdout: mon election default strategy = 1 2024-09-15T03:58:15.970 INFO:teuthology.orchestra.run.smithi110.stdout: ms bind msgr1 = False 2024-09-15T03:58:15.970 INFO:teuthology.orchestra.run.smithi110.stdout: ms bind msgr2 = True 2024-09-15T03:58:15.970 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject delay max = 1 2024-09-15T03:58:15.970 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject delay probability = 0.005 2024-09-15T03:58:15.970 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject delay type = mon 2024-09-15T03:58:15.971 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject internal delays = 0.002 2024-09-15T03:58:15.971 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject socket failures = 2500 2024-09-15T03:58:15.971 INFO:teuthology.orchestra.run.smithi110.stdout: ms type = async 2024-09-15T03:58:15.971 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:15.971 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.971 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2024-09-15T03:58:15.971 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2024-09-15T03:58:15.971 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.971 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2024-09-15T03:58:15.971 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 600 2024-09-15T03:58:15.971 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock profile = high_recovery_ops 2024-09-15T03:58:15.971 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.971 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2024-09-15T03:58:15.971 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2024-09-15T03:58:15.972 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.972 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2024-09-15T03:58:15.972 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2024-09-15T03:58:15.972 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2024-09-15T03:58:15.972 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.972 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2024-09-15T03:58:15.972 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2024-09-15T03:58:15.972 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.972 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2024-09-15T03:58:15.972 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.972 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2024-09-15T03:58:15.972 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.972 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2024-09-15T03:58:15.972 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore zero block detection = True 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 20 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify cached snaps = True 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify missing on start = True 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock override recovery settings = True 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: osd objectstore = filestore 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue = debug_random 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue cut off = debug_random 2024-09-15T03:58:15.973 INFO:teuthology.orchestra.run.smithi110.stdout: osd sloppy crc = True 2024-09-15T03:58:15.974 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.974 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2024-09-15T03:58:15.974 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-09-15T03:58:15.974 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2024-09-15T03:58:15.974 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-09-15T03:58:15.974 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-09-15T03:58:15.974 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:58:15.974 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:58:15.974 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2024-09-15T03:58:15.974 INFO:teuthology.orchestra.run.smithi110.stdout: debug monc = 10 2024-09-15T03:58:15.974 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.974 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2024-09-15T03:58:15.974 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.975 INFO:teuthology.orchestra.run.smithi110.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T03:58:15.976 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T03:58:15.976 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T03:58:15.976 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.976 INFO:teuthology.orchestra.run.smithi110.stdout: # 1m isn't quite enough 2024-09-15T03:58:15.976 INFO:teuthology.orchestra.run.smithi110.stdout: mon_down_mkfs_grace = 2m 2024-09-15T03:58:15.976 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.976 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_filestore_osds = false 2024-09-15T03:58:15.976 INFO:teuthology.orchestra.run.smithi110.stdout: mon min osdmap epochs = 25 2024-09-15T03:58:15.976 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune interval = 2 2024-09-15T03:58:15.976 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune min = 15 2024-09-15T03:58:15.976 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune txsize = 2 2024-09-15T03:58:15.976 INFO:teuthology.orchestra.run.smithi110.stdout: mon scrub inject crc mismatch = 0.01 2024-09-15T03:58:15.976 INFO:teuthology.orchestra.run.smithi110.stdout: mon scrub inject missing keys = 0.05 2024-09-15T03:58:15.976 INFO:teuthology.orchestra.run.smithi110.stdout: mon scrub interval = 300 2024-09-15T03:58:15.977 INFO:teuthology.orchestra.run.smithi110.stdout: paxos min = 10 2024-09-15T03:58:15.977 INFO:teuthology.orchestra.run.smithi110.stdout: paxos service trim min = 5 2024-09-15T03:58:15.977 INFO:teuthology.orchestra.run.smithi110.stdout: paxos trim min = 10 2024-09-15T03:58:15.977 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:15.977 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2024-09-15T03:58:15.977 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2024-09-15T03:58:15.977 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2024-09-15T03:58:15.977 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2024-09-15T03:58:15.977 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T03:58:15.977 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T03:58:15.977 INFO:teuthology.orchestra.run.smithi110.stdout: debug monc = 20 2024-09-15T03:58:15.977 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2024-09-15T03:58:15.977 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2024-09-15T03:58:15.977 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.c] 2024-09-15T03:58:15.978 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.b] 2024-09-15T03:58:15.978 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-09-15T03:58:15.978 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-09-15T03:58:15.978 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T03:58:15.978 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-09-15T03:58:15.979 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.979 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-09-15T03:58:15.979 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.979 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-09-15T03:58:15.979 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.979 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-09-15T03:58:15.979 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-09-15T03:58:15.979 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.979 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-09-15T03:58:15.979 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-09-15T03:58:15.979 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.979 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T03:58:15.979 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-09-15T03:58:15.980 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.980 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T03:58:15.980 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T03:58:15.980 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.980 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-09-15T03:58:15.980 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.980 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-09-15T03:58:15.980 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-09-15T03:58:15.980 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-09-15T03:58:15.980 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-09-15T03:58:15.980 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-09-15T03:58:15.980 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-09-15T03:58:15.980 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-09-15T03:58:15.980 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T03:58:15.981 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T03:58:15.981 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-09-15T03:58:15.981 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.981 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T03:58:15.981 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.981 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-09-15T03:58:15.981 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.981 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-09-15T03:58:15.981 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.981 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-09-15T03:58:15.981 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-09-15T03:58:15.982 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-09-15T03:58:15.982 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "172.21.15.110,[v2:172.21.15.110:3301,v1:172.21.15.110:6790],172.21.15.177" 2024-09-15T03:58:15.982 INFO:teuthology.orchestra.run.smithi177.stdout: mon client directed command retry = 5 2024-09-15T03:58:15.982 INFO:teuthology.orchestra.run.smithi177.stdout: mon election default strategy = 1 2024-09-15T03:58:15.982 INFO:teuthology.orchestra.run.smithi177.stdout: ms bind msgr1 = False 2024-09-15T03:58:15.982 INFO:teuthology.orchestra.run.smithi177.stdout: ms bind msgr2 = True 2024-09-15T03:58:15.982 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject delay max = 1 2024-09-15T03:58:15.982 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject delay probability = 0.005 2024-09-15T03:58:15.982 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject delay type = mon 2024-09-15T03:58:15.982 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject internal delays = 0.002 2024-09-15T03:58:15.982 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject socket failures = 2500 2024-09-15T03:58:15.982 INFO:teuthology.orchestra.run.smithi177.stdout: ms type = async 2024-09-15T03:58:15.982 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:15.983 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.983 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-09-15T03:58:15.983 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-09-15T03:58:15.983 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.983 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-09-15T03:58:15.983 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 600 2024-09-15T03:58:15.983 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-09-15T03:58:15.983 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.983 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-09-15T03:58:15.983 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-09-15T03:58:15.983 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.983 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-09-15T03:58:15.983 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-09-15T03:58:15.983 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-09-15T03:58:15.984 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.984 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-09-15T03:58:15.984 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-09-15T03:58:15.984 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.984 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-09-15T03:58:15.984 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.984 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-09-15T03:58:15.984 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.984 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-09-15T03:58:15.984 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.984 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-09-15T03:58:15.984 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-09-15T03:58:15.984 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.984 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-09-15T03:58:15.985 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-09-15T03:58:15.985 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore zero block detection = True 2024-09-15T03:58:15.985 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-15T03:58:15.985 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-09-15T03:58:15.985 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify cached snaps = True 2024-09-15T03:58:15.985 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify missing on start = True 2024-09-15T03:58:15.985 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock override recovery settings = True 2024-09-15T03:58:15.985 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = filestore 2024-09-15T03:58:15.985 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue = debug_random 2024-09-15T03:58:15.985 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue cut off = debug_random 2024-09-15T03:58:15.985 INFO:teuthology.orchestra.run.smithi177.stdout: osd sloppy crc = True 2024-09-15T03:58:15.985 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.985 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-09-15T03:58:15.985 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-15T03:58:15.986 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-09-15T03:58:15.986 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-09-15T03:58:15.986 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-09-15T03:58:15.986 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:58:15.986 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:58:15.986 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-09-15T03:58:15.986 INFO:teuthology.orchestra.run.smithi177.stdout: debug monc = 10 2024-09-15T03:58:15.986 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.986 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-09-15T03:58:15.986 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-15T03:58:15.986 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-09-15T03:58:15.986 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-09-15T03:58:15.986 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-09-15T03:58:15.986 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-09-15T03:58:15.987 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-09-15T03:58:15.987 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:58:15.987 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-09-15T03:58:15.987 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-09-15T03:58:15.987 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:58:15.987 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.987 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T03:58:15.987 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T03:58:15.987 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T03:58:15.987 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.987 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T03:58:15.987 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T03:58:15.987 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T03:58:15.987 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.988 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-09-15T03:58:15.988 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-09-15T03:58:15.988 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.988 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-09-15T03:58:15.988 INFO:teuthology.orchestra.run.smithi177.stdout: mon min osdmap epochs = 25 2024-09-15T03:58:15.988 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune interval = 2 2024-09-15T03:58:15.988 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune min = 15 2024-09-15T03:58:15.988 INFO:teuthology.orchestra.run.smithi177.stdout: mon osdmap full prune txsize = 2 2024-09-15T03:58:15.988 INFO:teuthology.orchestra.run.smithi177.stdout: mon scrub inject crc mismatch = 0.01 2024-09-15T03:58:15.988 INFO:teuthology.orchestra.run.smithi177.stdout: mon scrub inject missing keys = 0.05 2024-09-15T03:58:15.988 INFO:teuthology.orchestra.run.smithi177.stdout: mon scrub interval = 300 2024-09-15T03:58:15.988 INFO:teuthology.orchestra.run.smithi177.stdout: paxos min = 10 2024-09-15T03:58:15.988 INFO:teuthology.orchestra.run.smithi177.stdout: paxos service trim min = 5 2024-09-15T03:58:15.989 INFO:teuthology.orchestra.run.smithi177.stdout: paxos trim min = 10 2024-09-15T03:58:15.989 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-15T03:58:15.989 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-09-15T03:58:15.989 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-09-15T03:58:15.989 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-09-15T03:58:15.989 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-09-15T03:58:15.989 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T03:58:15.989 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T03:58:15.989 INFO:teuthology.orchestra.run.smithi177.stdout: debug monc = 20 2024-09-15T03:58:15.989 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-15T03:58:15.989 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-09-15T03:58:15.989 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-09-15T03:58:15.989 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-09-15T03:58:15.991 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-15T03:58:15.991 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-15T03:58:16.080 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-15T03:58:16.080 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:16.080 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-15T03:58:16.095 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:16.096 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-15T03:58:16.151 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2024-09-15T03:58:16.151 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:16.151 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T03:58:16.151 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:58:16.225 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:16.225 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:58:16.281 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi177.front.sepia.ceph.com 2024-09-15T03:58:16.281 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:16.281 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T03:58:16.281 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:58:16.319 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:16.319 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:58:16.375 INFO:tasks.ceph:Setting up mon nodes... 2024-09-15T03:58:16.376 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-15T03:58:16.376 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T03:58:16.470 INFO:teuthology.orchestra.run.smithi177.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T03:58:16.473 INFO:tasks.ceph:Setting up mds nodes... 2024-09-15T03:58:16.473 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-15T03:58:16.473 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-15T03:58:16.511 INFO:teuthology.orchestra.run.smithi177.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-15T03:58:16.523 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-15T03:58:16.523 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-15T03:58:16.523 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-15T03:58:16.548 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T03:58:16.548 INFO:tasks.ceph:role: osd.0 2024-09-15T03:58:16.548 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com 2024-09-15T03:58:16.548 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T03:58:16.612 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:58:16.613 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:58:16.613 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:58:16.613 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:58:16.613 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:58:16.613 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:58:16.613 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:58:16.613 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:58:16.613 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:58:16.613 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:58:16.646 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-09-15T03:58:16.648 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-09-15T03:58:16.648 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-15T03:58:16.717 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-15T03:58:16.786 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-15T03:58:16.851 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T03:58:16.852 INFO:tasks.ceph:role: osd.1 2024-09-15T03:58:16.852 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com 2024-09-15T03:58:16.852 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T03:58:16.917 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:58:16.917 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:58:16.917 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:58:16.917 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:58:16.917 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:58:16.917 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:58:16.917 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:58:16.917 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:58:16.917 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:58:16.917 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:58:16.949 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-09-15T03:58:16.951 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-09-15T03:58:16.951 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-15T03:58:17.020 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-15T03:58:17.087 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-15T03:58:17.151 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T03:58:17.151 INFO:tasks.ceph:role: osd.2 2024-09-15T03:58:17.152 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com 2024-09-15T03:58:17.152 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T03:58:17.215 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:58:17.215 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:58:17.216 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:58:17.216 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:58:17.216 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:58:17.216 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:58:17.216 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:58:17.216 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:58:17.216 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:58:17.216 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:58:17.248 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2024-09-15T03:58:17.250 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2024-09-15T03:58:17.250 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-15T03:58:17.319 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-15T03:58:17.387 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:58:17.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.479+0000 7fab0211f440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-15T03:58:17.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.480+0000 7fab0211f440 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-15T03:58:17.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.483+0000 7fab0211f440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:17.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.484+0000 7fab0211f440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:58:17.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.542+0000 7fab0211f440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:17.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.542+0000 7fab0211f440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:58:17.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.542+0000 7fab0211f440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:58:17.549 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-15T03:58:17.573 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:58:17.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.653+0000 7f97ce57b440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-15T03:58:17.654 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.653+0000 7f97ce57b440 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-15T03:58:17.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.657+0000 7f97ce57b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:17.658 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.657+0000 7f97ce57b440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:58:17.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.714+0000 7f97ce57b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:17.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.714+0000 7f97ce57b440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:58:17.715 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.714+0000 7f97ce57b440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:58:17.721 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-15T03:58:17.787 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:58:17.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.867+0000 7f913d92b440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-15T03:58:17.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.867+0000 7f913d92b440 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-15T03:58:17.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.871+0000 7f913d92b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:17.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.871+0000 7f913d92b440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:58:17.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.928+0000 7f913d92b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:17.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.928+0000 7f913d92b440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:58:17.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:17.928+0000 7f913d92b440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:58:17.936 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-15T03:58:18.002 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-15T03:58:18.031 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T03:58:18.031 INFO:tasks.ceph:role: osd.3 2024-09-15T03:58:18.032 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-15T03:58:18.032 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T03:58:18.107 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:58:18.108 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:58:18.108 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:58:18.108 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:58:18.108 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:58:18.108 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:58:18.108 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:58:18.108 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:58:18.108 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:58:18.108 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:58:18.604 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-09-15T03:58:18.606 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-09-15T03:58:18.607 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-15T03:58:18.719 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-15T03:58:18.753 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-15T03:58:18.821 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T03:58:18.821 INFO:tasks.ceph:role: osd.4 2024-09-15T03:58:18.821 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-15T03:58:18.821 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T03:58:18.895 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:58:18.895 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:58:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:58:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:58:18.896 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:58:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:58:18.896 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:58:18.896 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:58:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:58:18.896 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:58:19.362 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-09-15T03:58:19.365 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-09-15T03:58:19.366 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-15T03:58:19.459 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-15T03:58:19.488 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-15T03:58:19.554 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T03:58:19.554 INFO:tasks.ceph:role: osd.5 2024-09-15T03:58:19.554 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-15T03:58:19.554 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T03:58:19.626 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:58:19.626 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:58:19.626 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:58:19.626 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T03:58:19.627 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:58:19.627 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:58:19.627 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:58:19.627 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T03:58:19.627 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:58:19.627 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:58:20.100 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-09-15T03:58:20.102 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-09-15T03:58:20.102 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-15T03:58:20.240 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-15T03:58:20.269 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:58:20.366 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:20.365+0000 7f3d5b232440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-15T03:58:20.370 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:20.370+0000 7f3d5b232440 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-15T03:58:20.395 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:20.395+0000 7f3d5b232440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:20.396 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:20.395+0000 7f3d5b232440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:58:20.640 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:20.639+0000 7f3d5b232440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:20.644 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:20.644+0000 7f3d5b232440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:58:20.644 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:20.644+0000 7f3d5b232440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:58:20.669 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-15T03:58:20.694 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:58:20.775 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:20.775+0000 7f9d32a0c440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-15T03:58:20.781 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:20.780+0000 7f9d32a0c440 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-15T03:58:20.809 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:20.809+0000 7f9d32a0c440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:20.811 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:20.811+0000 7f9d32a0c440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:58:21.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:21.038+0000 7f9d32a0c440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:21.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:21.038+0000 7f9d32a0c440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:58:21.039 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:21.038+0000 7f9d32a0c440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:58:21.059 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-15T03:58:21.125 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:58:21.206 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:21.205+0000 7f68f020b440 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-15T03:58:21.206 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:21.206+0000 7f68f020b440 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-15T03:58:21.210 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:21.210+0000 7f68f020b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:21.211 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:21.210+0000 7f68f020b440 -1 journal FileJournal::_open_file : zeroing journal 2024-09-15T03:58:21.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:21.387+0000 7f68f020b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:21.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:21.387+0000 7f68f020b440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:58:21.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:21.387+0000 7f68f020b440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T03:58:21.395 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-15T03:58:21.460 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-15T03:58:21.460 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:21.460 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-15T03:58:21.490 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:21.491 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-15T03:58:21.559 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:21.560 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-15T03:58:21.627 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:21.628 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-15T03:58:21.653 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:21.653 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-15T03:58:21.718 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:21.718 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-15T03:58:21.783 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:21.784 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-15T03:58:21.849 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:21.849 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-15T03:58:21.906 INFO:tasks.ceph:Adding keys to all mons... 2024-09-15T03:58:21.906 DEBUG:teuthology.orchestra.run.smithi110:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T03:58:21.908 DEBUG:teuthology.orchestra.run.smithi177:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T03:58:21.949 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2024-09-15T03:58:21.949 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQBZW+ZmsGOpHBAA0wv1rcIQqj7S8Mk+pTdGMQ== 2024-09-15T03:58:21.949 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2024-09-15T03:58:21.949 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQBZW+ZmzcX+JhAAyYgyDLi0804/Z6AAKCoxaw== 2024-09-15T03:58:21.949 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2024-09-15T03:58:21.950 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQBZW+ZmJf/FMxAAD4u+gV3GCFoGFxVx5tu8zQ== 2024-09-15T03:58:21.950 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.x] 2024-09-15T03:58:21.950 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQBYW+ZmG/oRHBAAubRCS2tL+gwqzZ0Mabpn+A== 2024-09-15T03:58:21.950 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.3] 2024-09-15T03:58:21.950 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQBcW+Zm5ljUFRAAHaxcoJD7g9vA+Xgpn80l2Q== 2024-09-15T03:58:21.950 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.4] 2024-09-15T03:58:21.950 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQBcW+ZmeMY9LhAA0rkQ3ijMJz0FfkyigwB5aw== 2024-09-15T03:58:21.950 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.5] 2024-09-15T03:58:21.950 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQBdW+ZmwHlNDBAAwZJ7HqDOykXYyM4TUix9yA== 2024-09-15T03:58:21.950 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2024-09-15T03:58:21.950 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQBYW+ZmOll6HhAAn6qIsl5/Q15P8X6VQDr8IA== 2024-09-15T03:58:21.969 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.0] 2024-09-15T03:58:21.969 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBZW+ZmsGOpHBAA0wv1rcIQqj7S8Mk+pTdGMQ== 2024-09-15T03:58:21.970 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.1] 2024-09-15T03:58:21.970 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBZW+ZmzcX+JhAAyYgyDLi0804/Z6AAKCoxaw== 2024-09-15T03:58:21.970 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.2] 2024-09-15T03:58:21.970 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBZW+ZmJf/FMxAAD4u+gV3GCFoGFxVx5tu8zQ== 2024-09-15T03:58:21.970 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.x] 2024-09-15T03:58:21.970 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBYW+ZmG/oRHBAAubRCS2tL+gwqzZ0Mabpn+A== 2024-09-15T03:58:21.970 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.3] 2024-09-15T03:58:21.970 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBcW+Zm5ljUFRAAHaxcoJD7g9vA+Xgpn80l2Q== 2024-09-15T03:58:21.970 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.4] 2024-09-15T03:58:21.970 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBcW+ZmeMY9LhAA0rkQ3ijMJz0FfkyigwB5aw== 2024-09-15T03:58:21.971 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.5] 2024-09-15T03:58:21.971 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBdW+ZmwHlNDBAAwZJ7HqDOykXYyM4TUix9yA== 2024-09-15T03:58:21.971 INFO:teuthology.orchestra.run.smithi177.stdout:[client.0] 2024-09-15T03:58:21.971 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBYW+ZmOll6HhAAn6qIsl5/Q15P8X6VQDr8IA== 2024-09-15T03:58:21.972 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:58:21.994 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:58:22.058 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:58:22.061 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:58:22.107 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:58:22.110 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:58:22.156 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:58:22.158 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:58:22.244 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:58:22.247 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:58:22.294 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:58:22.297 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:58:22.343 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:58:22.345 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:58:22.392 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T03:58:22.396 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T03:58:22.444 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-15T03:58:22.444 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-15T03:58:22.468 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:58:22.559 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-15T03:58:22.583 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-15T03:58:22.648 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:58:22.739 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-15T03:58:22.764 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-15T03:58:22.790 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:58:22.881 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-15T03:58:22.906 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:58:22.908 DEBUG:teuthology.orchestra.run.smithi177:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:58:22.961 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-15T03:58:22.962 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:58:22.962 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:58:22.964 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:58:22.965 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:58:22.965 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:58:22.967 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:58:22.967 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:58:22.967 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:58:23.004 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:58:23.004 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-15T03:58:23.004 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-15T03:58:23.004 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-15T03:58:23.006 INFO:tasks.ceph.mgr.x:Started 2024-09-15T03:58:23.007 INFO:tasks.ceph:Setting crush tunables to default 2024-09-15T03:58:23.007 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd crush tunables default 2024-09-15T03:58:23.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.099+0000 7fdbdbaf8640 10 monclient: get_monmap_and_config 2024-09-15T03:58:23.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.099+0000 7fdbdbaf8640 10 monclient: build_initial_monmap 2024-09-15T03:58:23.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.099+0000 7fdbdbaf8640 10 monclient: monmap: 2024-09-15T03:58:23.100 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:23.100 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:23.100 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:23.100094+0000 2024-09-15T03:58:23.100 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:23.100094+0000 2024-09-15T03:58:23.100 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:23.100 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:23.100 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:23.100 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:23.101 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:23.101 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:23.101 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:23.101 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:23.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.099+0000 7fdbdbaf8640 1 Processor -- start 2024-09-15T03:58:23.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.099+0000 7fdbdbaf8640 1 -- start start 2024-09-15T03:58:23.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.099+0000 7fdbdbaf8640 10 monclient: init 2024-09-15T03:58:23.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbdbaf8640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:23.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbdbaf8640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T03:58:23.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbdbaf8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 0x7fdbd4131ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:23.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbdbaf8640 10 monclient(hunting): picked mon.noname-b con 0x7fdbd40a17f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:23.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbdbaf8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdbd409e940 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:23.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbdbaf8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 0x7fdbd409fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:23.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbdbaf8640 10 monclient(hunting): picked mon.noname-a con 0x7fdbd409ed70 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:23.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbdbaf8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd40a1bc0 con 0x7fdbd409e940 2024-09-15T03:58:23.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbdbaf8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:23.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbdbaf8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbd409d2e0 con 0x7fdbd409ed70 2024-09-15T03:58:23.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbdbaf8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:23.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbdbaf8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd409c700 con 0x7fdbd40a17f0 2024-09-15T03:58:23.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbdbaf8640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:23.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbdbaf8640 10 monclient(hunting): authenticate will time out at 726.744452s 2024-09-15T03:58:23.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbd9093640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fdbd409e940 0x7fdbd409f760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51880/0 (socket says 172.21.15.110:51880) 2024-09-15T03:58:23.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbd9093640 1 -- 172.21.15.110:0/1587018194 learned_addr learned my addr 172.21.15.110:0/1587018194 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:23.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 0x7fdbd4131ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:23.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbda095640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 0x7fdbd409fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:23.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbd9894640 10 monclient(hunting): get_auth_request con 0x7fdbd40a17f0 auth_method 0 2024-09-15T03:58:23.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbd9894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:23.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbd9894640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:23.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbd9894640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:23.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbda095640 10 monclient(hunting): get_auth_request con 0x7fdbd409ed70 auth_method 0 2024-09-15T03:58:23.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbda095640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:23.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbda095640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:23.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbda095640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:23.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 msgr2=0x7fdbd4131ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:58:23.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 msgr2=0x7fdbd4131ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:58:23.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 0x7fdbd4131ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:58:23.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbda095640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 msgr2=0x7fdbd409fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:58:23.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbda095640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 msgr2=0x7fdbd409fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:58:23.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbda095640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 0x7fdbd409fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:58:23.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 0x7fdbd4131ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:58:23.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.100+0000 7fdbda095640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 0x7fdbd409fe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:58:23.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.102+0000 7fdbd9093640 1 -- 172.21.15.110:0/1587018194 >> v1:172.21.15.110:6789/0 conn(0x7fdbd409e940 legacy=0x7fdbd409f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:23.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 0x7fdbd4131ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:23.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbda095640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 0x7fdbd409fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:23.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbd9894640 10 monclient(hunting): get_auth_request con 0x7fdbd40a17f0 auth_method 0 2024-09-15T03:58:23.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbd9894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:23.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbd9894640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:23.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbd9894640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:58:23.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbda095640 10 monclient(hunting): get_auth_request con 0x7fdbd409ed70 auth_method 0 2024-09-15T03:58:23.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbda095640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:23.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbda095640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:23.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbda095640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:58:23.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 msgr2=0x7fdbd4131ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:58:23.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 msgr2=0x7fdbd4131ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:58:23.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbda095640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 msgr2=0x7fdbd409fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:58:23.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbda095640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 msgr2=0x7fdbd409fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:58:23.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 0x7fdbd4131ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:58:23.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbda095640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 0x7fdbd409fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:58:23.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 0x7fdbd4131ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:58:23.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.301+0000 7fdbda095640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 0x7fdbd409fe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:58:23.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 0x7fdbd4131ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:23.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbda095640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 0x7fdbd409fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:23.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbd9894640 10 monclient(hunting): get_auth_request con 0x7fdbd40a17f0 auth_method 0 2024-09-15T03:58:23.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbd9894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:23.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbd9894640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:23.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbd9894640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:58:23.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbda095640 10 monclient(hunting): get_auth_request con 0x7fdbd409ed70 auth_method 0 2024-09-15T03:58:23.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbda095640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:23.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbda095640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:23.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbda095640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:58:23.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 msgr2=0x7fdbd4131ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:58:23.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbda095640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 msgr2=0x7fdbd409fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:58:23.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 msgr2=0x7fdbd4131ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:58:23.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbda095640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 msgr2=0x7fdbd409fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:58:23.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 0x7fdbd4131ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:58:23.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbda095640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 0x7fdbd409fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:58:23.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbda095640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 0x7fdbd409fe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:58:23.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:23.702+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 0x7fdbd4131ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:58:24.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbda095640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 0x7fdbd409fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:24.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbda095640 10 monclient(hunting): get_auth_request con 0x7fdbd409ed70 auth_method 0 2024-09-15T03:58:24.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbda095640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:24.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbda095640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:24.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 0x7fdbd4131ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:24.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbda095640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:58:24.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbd9894640 10 monclient(hunting): get_auth_request con 0x7fdbd40a17f0 auth_method 0 2024-09-15T03:58:24.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbd9894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:24.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbd9894640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:24.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbd9894640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:58:24.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbda095640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 msgr2=0x7fdbd409fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:58:24.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbda095640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 msgr2=0x7fdbd409fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:58:24.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbda095640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 0x7fdbd409fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:58:24.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbda095640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 0x7fdbd409fe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:58:24.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 msgr2=0x7fdbd4131ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:58:24.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 msgr2=0x7fdbd4131ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:58:24.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 0x7fdbd4131ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:58:24.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:24.503+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 0x7fdbd4131ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:58:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 10 monclient(hunting): tick 2024-09-15T03:58:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 1 monclient(hunting): continuing hunt 2024-09-15T03:58:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:58:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 1 -- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 msgr2=0x7fdbd4131ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:58:26.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbd40a17f0 0x7fdbd4131ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 msgr2=0x7fdbd409fe70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:58:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.110:3300/0 conn(0x7fdbd409ed70 0x7fdbd409fe70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 1 -- 172.21.15.110:0/1587018194 >> v1:172.21.15.110:6789/0 conn(0x7fdbd409e940 legacy=0x7fdbd409f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 10 monclient: _add_conns ranks=[4,1] 2024-09-15T03:58:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 0x7fdbb8005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 10 monclient(hunting): picked mon.noname-c con 0x7fdbb8003720 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbb80061c0 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 1 -- 172.21.15.110:0/1587018194 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd41320d0 con 0x7fdbb80061c0 2024-09-15T03:58:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 1 -- 172.21.15.110:0/1587018194 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbd409d2e0 con 0x7fdbb8003720 2024-09-15T03:58:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbcb7fe640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 0x7fdbb8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbd9894640 10 monclient(hunting): get_auth_request con 0x7fdbb8003720 auth_method 0 2024-09-15T03:58:26.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbd9894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbd9894640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbd9894640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 msgr2=0x7fdbb8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:58:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 msgr2=0x7fdbb8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:58:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 0x7fdbb8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:58:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.101+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 0x7fdbb8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:58:26.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.103+0000 7fdbd9093640 1 -- 172.21.15.110:0/1587018194 >> v1:172.21.15.177:6789/0 conn(0x7fdbb80061c0 legacy=0x7fdbb8008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:26.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.302+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 0x7fdbb8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:26.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.302+0000 7fdbd9894640 10 monclient(hunting): get_auth_request con 0x7fdbb8003720 auth_method 0 2024-09-15T03:58:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.302+0000 7fdbd9894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.302+0000 7fdbd9894640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.302+0000 7fdbd9894640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:58:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.302+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 msgr2=0x7fdbb8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:58:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.302+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 msgr2=0x7fdbb8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:58:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.302+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 0x7fdbb8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:58:26.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.302+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 0x7fdbb8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:58:26.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.703+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 0x7fdbb8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:26.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.703+0000 7fdbd9894640 10 monclient(hunting): get_auth_request con 0x7fdbb8003720 auth_method 0 2024-09-15T03:58:26.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.703+0000 7fdbd9894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:26.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.703+0000 7fdbd9894640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:26.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.703+0000 7fdbd9894640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:58:26.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.703+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 msgr2=0x7fdbb8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:58:26.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.703+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 msgr2=0x7fdbb8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:58:26.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.703+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 0x7fdbb8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:58:26.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:26.703+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 0x7fdbb8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:58:27.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:27.504+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 0x7fdbb8005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:27.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:27.504+0000 7fdbd9894640 10 monclient(hunting): get_auth_request con 0x7fdbb8003720 auth_method 0 2024-09-15T03:58:27.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:27.504+0000 7fdbd9894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:27.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:27.504+0000 7fdbd9894640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:27.504+0000 7fdbd9894640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:58:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:27.504+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 msgr2=0x7fdbb8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:58:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:27.504+0000 7fdbd9894640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 msgr2=0x7fdbb8005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:58:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:27.504+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 0x7fdbb8005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:58:27.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:27.504+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 0x7fdbb8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:58:28.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.619+0000 7fdbd8892640 1 -- 172.21.15.110:0/1587018194 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377306272 0 0) 0x7fdbd41320d0 con 0x7fdbb80061c0 2024-09-15T03:58:28.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.619+0000 7fdbd8892640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:28.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.619+0000 7fdbd8892640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:28.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.619+0000 7fdbd8892640 10 monclient(hunting): my global_id is 4099 2024-09-15T03:58:28.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.619+0000 7fdbd8892640 1 -- 172.21.15.110:0/1587018194 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc00034e0 con 0x7fdbb80061c0 2024-09-15T03:58:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 1 -- 172.21.15.110:0/1587018194 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3620842395 0 0) 0x7fdbc00034e0 con 0x7fdbb80061c0 2024-09-15T03:58:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 1 -- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 msgr2=0x7fdbb8005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:58:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 1 --2- 172.21.15.110:0/1587018194 >> v2:172.21.15.177:3300/0 conn(0x7fdbb8003720 0x7fdbb8005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 10 monclient: _finish_hunting 0 2024-09-15T03:58:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:58:28.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.177:6789/0 2024-09-15T03:58:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 1 -- 172.21.15.110:0/1587018194 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb800af40 con 0x7fdbb80061c0 2024-09-15T03:58:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 10 monclient: _finish_auth 0 2024-09-15T03:58:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 1 -- 172.21.15.110:0/1587018194 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3165669920 0 0) 0x7fdbc4001720 con 0x7fdbb80061c0 2024-09-15T03:58:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbdbaf8640 5 monclient: authenticate success, global_id 4099 2024-09-15T03:58:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbdbaf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:58:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 10 monclient: dump: 2024-09-15T03:58:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:28.625 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:28.626 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] but i'm connected to v1:172.21.15.177:6789/0, reconnecting 2024-09-15T03:58:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 1 -- 172.21.15.110:0/1587018194 >> v1:172.21.15.177:6789/0 conn(0x7fdbb80061c0 legacy=0x7fdbb8008650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:28.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:58:28.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbda095640 1 -- 172.21.15.110:0/1587018194 reap_dead start 2024-09-15T03:58:28.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc0003a20 0x7fdbc0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:28.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 10 monclient(hunting): picked mon.b con 0x7fdbc0003a20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:28.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc0004780 0x7fdbc0004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:28.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.620+0000 7fdbd8892640 10 monclient(hunting): picked mon.c con 0x7fdbc0004780 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:28.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd8892640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc0005330 0x7fdbc000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:28.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd8892640 10 monclient(hunting): picked mon.a con 0x7fdbc0005330 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:28.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd8892640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:28.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd8892640 1 -- 172.21.15.110:0/1587018194 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd409d2e0 con 0x7fdbc0005330 2024-09-15T03:58:28.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd8892640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:28.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd8892640 1 -- 172.21.15.110:0/1587018194 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc000c260 con 0x7fdbc0003a20 2024-09-15T03:58:28.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd8892640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:28.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd8892640 1 -- 172.21.15.110:0/1587018194 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc000c420 con 0x7fdbc0004780 2024-09-15T03:58:28.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd8892640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:28.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd8892640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:58:28.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbdbaf8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:28.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc0004780 0x7fdbc0004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:28.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 10 monclient(hunting): get_auth_request con 0x7fdbc0004780 auth_method 0 2024-09-15T03:58:28.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:28.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:28.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:58:28.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbda095640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc0005330 0x7fdbc000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:58:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbda095640 10 monclient(hunting): get_auth_request con 0x7fdbc0005330 auth_method 0 2024-09-15T03:58:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbda095640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbda095640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbda095640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:58:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbda095640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbda095640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:28.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbda095640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:58:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 10 monclient(hunting): handle_auth_done global_id 4099 payload 293 2024-09-15T03:58:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 1 -- 172.21.15.110:0/1587018194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc0003a20 msgr2=0x7fdbc0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc0003a20 0x7fdbc0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 1 -- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc0005330 msgr2=0x7fdbc000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc0005330 0x7fdbc000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 10 monclient: _finish_hunting 0 2024-09-15T03:58:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 1 monclient: found mon.c 2024-09-15T03:58:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 1 -- 172.21.15.110:0/1587018194 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc00034e0 con 0x7fdbc0004780 2024-09-15T03:58:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 10 monclient: _finish_auth 0 2024-09-15T03:58:28.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.621+0000 7fdbd9093640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:28.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbd9093640 1 -- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc0004780 msgr2=0x7fdbc0004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:28.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbd9093640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc0004780 0x7fdbc0004bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40014f0 tx=0x7fdbc40079f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1d90ea25b3dd408 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:28.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbd8892640 1 -- 172.21.15.110:0/1587018194 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4005890 con 0x7fdbc0004780 2024-09-15T03:58:28.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbd8892640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:28.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbd8892640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:28.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbd8892640 10 monclient: dump: 2024-09-15T03:58:28.637 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:28.638 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:28.638 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:28.638 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:28.638 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:28.638 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:28.638 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:28.638 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:28.638 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:28.638 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:28.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbd8892640 1 -- 172.21.15.110:0/1587018194 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc4005a10 con 0x7fdbc0004780 2024-09-15T03:58:28.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbdbaf8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:28.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbd8892640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:28.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbdbaf8640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:28.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbd8892640 1 -- 172.21.15.110:0/1587018194 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4005dd0 con 0x7fdbc0004780 2024-09-15T03:58:28.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbdbaf8640 4 set_mon_vals no callback set 2024-09-15T03:58:28.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbd8892640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:28.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbd8892640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:28.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.624+0000 7fdbd8892640 10 monclient: dump: 2024-09-15T03:58:28.639 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:28.639 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:28.639 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:28.639 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:28.639 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:28.639 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:28.640 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:28.640 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:28.640 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:28.640 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:28.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.625+0000 7fdbdbaf8640 10 monclient: shutdown 2024-09-15T03:58:28.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.625+0000 7fdbdbaf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:28.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.625+0000 7fdbdbaf8640 1 -- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc0004780 msgr2=0x7fdbc0004bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:28.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.625+0000 7fdbdbaf8640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc0004780 0x7fdbc0004bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40014f0 tx=0x7fdbc40079f0 comp rx=0 tx=0).stop 2024-09-15T03:58:28.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.625+0000 7fdbdbaf8640 1 -- 172.21.15.110:0/1587018194 shutdown_connections 2024-09-15T03:58:28.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.625+0000 7fdbdbaf8640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc0005330 0x7fdbc000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:28.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.625+0000 7fdbdbaf8640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc0004780 0x7fdbc0004bf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:28.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.625+0000 7fdbdbaf8640 1 --2- 172.21.15.110:0/1587018194 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc0003a20 0x7fdbc0003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:28.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.625+0000 7fdbdbaf8640 1 -- 172.21.15.110:0/1587018194 >> 172.21.15.110:0/1587018194 conn(0x7fdbd4062dd0 msgr2=0x7fdbd40631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:28.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.625+0000 7fdbdbaf8640 1 -- 172.21.15.110:0/1587018194 shutdown_connections 2024-09-15T03:58:28.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.625+0000 7fdbdbaf8640 1 -- 172.21.15.110:0/1587018194 wait complete. 2024-09-15T03:58:28.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.626+0000 7fdbdbaf8640 10 monclient: build_initial_monmap 2024-09-15T03:58:28.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.626+0000 7fdbdbaf8640 10 monclient: monmap: 2024-09-15T03:58:28.641 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:28.641 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:28.641 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:28.641 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:28.641 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:28.642 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:28.642 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:28.642 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:28.642 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:28.642 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:28.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.626+0000 7fdbdbaf8640 1 Processor -- start 2024-09-15T03:58:28.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.626+0000 7fdbdbaf8640 1 -- start start 2024-09-15T03:58:28.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.626+0000 7fdbdbaf8640 10 monclient: init 2024-09-15T03:58:28.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:28.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:58:28.642 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc0003a20 0x7fdbd409a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:28.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc0003a20 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:28.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc0004780 0x7fdbd41304a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:28.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc0004780 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:28.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc0005330 0x7fdbd411b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:28.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc0005330 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:28.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:28.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd409b140 con 0x7fdbc0004780 2024-09-15T03:58:28.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:28.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd409c480 con 0x7fdbc0003a20 2024-09-15T03:58:28.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:28.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd40a5920 con 0x7fdbc0005330 2024-09-15T03:58:28.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:28.643 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbdbaf8640 10 monclient(hunting): authenticate will time out at 732.271337s 2024-09-15T03:58:28.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc0004780 0x7fdbd41304a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:28.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc0004780 0x7fdbd41304a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:55122/0 (socket says 172.21.15.110:55122) 2024-09-15T03:58:28.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 1 -- 172.21.15.110:0/1033989136 learned_addr learned my addr 172.21.15.110:0/1033989136 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:28.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 10 monclient(hunting): get_auth_request con 0x7fdbc0004780 auth_method 0 2024-09-15T03:58:28.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:28.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:28.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:28.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:28.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:28.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9894640 1 --2- 172.21.15.110:0/1033989136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc0003a20 0x7fdbd409a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:28.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:28.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9894640 10 monclient(hunting): get_auth_request con 0x7fdbc0003a20 auth_method 0 2024-09-15T03:58:28.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:28.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9894640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:28.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9894640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:28.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9894640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:28.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9894640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:28.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9894640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:28.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 10 monclient(hunting): handle_auth_done global_id 4101 payload 824 2024-09-15T03:58:28.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 1 -- 172.21.15.110:0/1033989136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc0005330 msgr2=0x7fdbd411b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:28.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 1 --2- 172.21.15.110:0/1033989136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc0005330 0x7fdbd411b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:28.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 1 -- 172.21.15.110:0/1033989136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc0003a20 msgr2=0x7fdbd409a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:28.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 1 --2- 172.21.15.110:0/1033989136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc0003a20 0x7fdbd409a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 10 monclient: _finish_hunting 0 2024-09-15T03:58:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 1 monclient: found mon.noname-a 2024-09-15T03:58:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.627+0000 7fdbd9093640 1 -- 172.21.15.110:0/1033989136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd411c6d0 con 0x7fdbc0004780 2024-09-15T03:58:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.630+0000 7fdbd9093640 1 -- 172.21.15.110:0/1033989136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc0004780 msgr2=0x7fdbd41304a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.630+0000 7fdbd9093640 1 --2- 172.21.15.110:0/1033989136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc0004780 0x7fdbd41304a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdbc402af20 tx=0x7fdbc4034000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.631+0000 7fdbc9ffb640 1 -- 172.21.15.110:0/1033989136 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4007720 con 0x7fdbc0004780 2024-09-15T03:58:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.631+0000 7fdbc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.631+0000 7fdbc9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.631+0000 7fdbc9ffb640 10 monclient: dump: 2024-09-15T03:58:28.646 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:28.647 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.631+0000 7fdbc9ffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:28.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.631+0000 7fdbc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:28.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.631+0000 7fdbdbaf8640 5 monclient: authenticate success, global_id 4101 2024-09-15T03:58:28.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.631+0000 7fdbdbaf8640 10 monclient: _renew_subs 2024-09-15T03:58:28.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.631+0000 7fdbdbaf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:28.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.631+0000 7fdbdbaf8640 1 -- 172.21.15.110:0/1033989136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd411d5c0 con 0x7fdbc0004780 2024-09-15T03:58:28.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.631+0000 7fdbdbaf8640 10 monclient: _renew_subs 2024-09-15T03:58:28.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.631+0000 7fdbdbaf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:28.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.631+0000 7fdbdbaf8640 1 -- 172.21.15.110:0/1033989136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd4135d70 con 0x7fdbc0004780 2024-09-15T03:58:28.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.632+0000 7fdbc9ffb640 1 -- 172.21.15.110:0/1033989136 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc4000c90 con 0x7fdbc0004780 2024-09-15T03:58:28.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.632+0000 7fdbc9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:28.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.632+0000 7fdbc9ffb640 1 -- 172.21.15.110:0/1033989136 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc403b6c0 con 0x7fdbc0004780 2024-09-15T03:58:28.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.632+0000 7fdbc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:28.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.632+0000 7fdbc9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:28.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.632+0000 7fdbc9ffb640 10 monclient: dump: 2024-09-15T03:58:28.649 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:28.649 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:28.649 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:28.649 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:28.649 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:28.649 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:28.649 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:28.649 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:28.649 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:28.649 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:28.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.632+0000 7fdbcbfff640 4 set_mon_vals no callback set 2024-09-15T03:58:28.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.632+0000 7fdbc9ffb640 1 -- 172.21.15.110:0/1033989136 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fdbc403bcc0 con 0x7fdbc0004780 2024-09-15T03:58:28.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.632+0000 7fdbc9ffb640 1 -- 172.21.15.110:0/1033989136 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fdbc4047870 con 0x7fdbc0004780 2024-09-15T03:58:28.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.633+0000 7fdbdbaf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:28.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.633+0000 7fdbdbaf8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:28.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.633+0000 7fdbdbaf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:28.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.633+0000 7fdbdbaf8640 1 -- 172.21.15.110:0/1033989136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb4005010 con 0x7fdbc0004780 2024-09-15T03:58:28.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.635+0000 7fdbc9ffb640 1 -- 172.21.15.110:0/1033989136 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7fdbc4047460 con 0x7fdbc0004780 2024-09-15T03:58:28.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.635+0000 7fdbc9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:28.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.635+0000 7fdbc9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:28.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.696+0000 7fdbdbaf8640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T03:58:28.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.696+0000 7fdbdbaf8640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T03:58:28.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.696+0000 7fdbdbaf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:28.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.696+0000 7fdbdbaf8640 1 -- 172.21.15.110:0/1033989136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fdbb4005590 con 0x7fdbc0004780 2024-09-15T03:58:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.699+0000 7fdbc9ffb640 1 -- 172.21.15.110:0/1033989136 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fdbc4047ad0 con 0x7fdbc0004780 2024-09-15T03:58:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.699+0000 7fdbc9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T03:58:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.699+0000 7fdbc9ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-15T03:58:28.700 INFO:teuthology.orchestra.run.smithi110.stderr:adjusted tunables profile to default 2024-09-15T03:58:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.700+0000 7fdbdbaf8640 10 monclient: shutdown 2024-09-15T03:58:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.700+0000 7fdbdbaf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.700+0000 7fdbdbaf8640 1 -- 172.21.15.110:0/1033989136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc0004780 msgr2=0x7fdbd41304a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.700+0000 7fdbdbaf8640 1 --2- 172.21.15.110:0/1033989136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc0004780 0x7fdbd41304a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdbc402af20 tx=0x7fdbc4034000 comp rx=0 tx=0).stop 2024-09-15T03:58:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.700+0000 7fdbdbaf8640 1 -- 172.21.15.110:0/1033989136 shutdown_connections 2024-09-15T03:58:28.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.700+0000 7fdbdbaf8640 1 --2- 172.21.15.110:0/1033989136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fdbc0005330 0x7fdbd411b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.700+0000 7fdbdbaf8640 1 --2- 172.21.15.110:0/1033989136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdbc0004780 0x7fdbd41304a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.700+0000 7fdbdbaf8640 1 --2- 172.21.15.110:0/1033989136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbc0003a20 0x7fdbd409a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.700+0000 7fdbdbaf8640 1 -- 172.21.15.110:0/1033989136 >> 172.21.15.110:0/1033989136 conn(0x7fdbd4062dd0 msgr2=0x7fdbd4134630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.700+0000 7fdbdbaf8640 1 -- 172.21.15.110:0/1033989136 shutdown_connections 2024-09-15T03:58:28.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:28.700+0000 7fdbdbaf8640 1 -- 172.21.15.110:0/1033989136 wait complete. 2024-09-15T03:58:28.712 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-15T03:58:28.712 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:28.712 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-15T03:58:28.731 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:28.730+0000 7fab3c215ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T03:58:28.740 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:28.740 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-15T03:58:28.770 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:28.769+0000 7fab3c215ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T03:58:28.805 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-09-15T03:58:28.805 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-15T03:58:28.819 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:28.817+0000 7fab3c215ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T03:58:28.870 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:28.870 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-15T03:58:28.895 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:28.895 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-15T03:58:28.960 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-15T03:58:28.960 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-15T03:58:28.961 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:28.960+0000 7fab3c215ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T03:58:28.985 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 105a3b3d-e410-47fb-ac54-6a5d5da4a46f 0 2024-09-15T03:58:29.073 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:29.072+0000 7fab3c215ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T03:58:29.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.117+0000 7f94ff8c9640 10 monclient: get_monmap_and_config 2024-09-15T03:58:29.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.117+0000 7f94ff8c9640 10 monclient: build_initial_monmap 2024-09-15T03:58:29.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.117+0000 7f94ff8c9640 10 monclient: monmap: 2024-09-15T03:58:29.119 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-09-15T03:58:29.119 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.119 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:29.119154+0000 2024-09-15T03:58:29.119 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:29.119154+0000 2024-09-15T03:58:29.119 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:29.119 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.119 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:29.119 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:29.119 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:29.120 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:29.120 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:29.120 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 1 Processor -- start 2024-09-15T03:58:29.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 1 -- start start 2024-09-15T03:58:29.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 10 monclient: init 2024-09-15T03:58:29.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:29.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-15T03:58:29.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f94f809dc10 0x7f94f809d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 10 monclient(hunting): picked mon.noname-c con 0x7f94f809dc10 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:29.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94f809dfe0 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:29.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94f81346c0 0x7f94f8136aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 10 monclient(hunting): picked mon.noname-b con 0x7f94f81346c0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:29.132 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f806da90 con 0x7f94f809dfe0 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f806c660 con 0x7f94f809dc10 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f809f2e0 con 0x7f94f81346c0 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94ff8c9640 10 monclient(hunting): authenticate will time out at 733.170879s 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.118+0000 7f94fd665640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f94f809dc10 0x7f94f809d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f94f809dc10 0x7f94f809d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:56106/0 (socket says 172.21.15.177:56106) 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 1 -- 172.21.15.177:0/2632579775 learned_addr learned my addr 172.21.15.177:0/2632579775 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 10 monclient(hunting): get_auth_request con 0x7f94f809dc10 auth_method 0 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fde66640 1 --2- 172.21.15.177:0/2632579775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94f81346c0 0x7f94f8136aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.133 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.134 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.134 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.134 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fde66640 10 monclient(hunting): get_auth_request con 0x7f94f81346c0 auth_method 0 2024-09-15T03:58:29.134 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fde66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.134 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fde66640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.134 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fde66640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.134 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fde66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.134 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fde66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.134 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fde66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.134 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 10 monclient(hunting): handle_auth_done global_id 4108 payload 274 2024-09-15T03:58:29.134 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 1 -- 172.21.15.177:0/2632579775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94f81346c0 msgr2=0x7f94f8136aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:29.134 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 1 --2- 172.21.15.177:0/2632579775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94f81346c0 0x7f94f8136aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.134 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 1 -- 172.21.15.177:0/2632579775 >> v1:172.21.15.110:6789/0 conn(0x7f94f809dfe0 legacy=0x7f94f809e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 10 monclient: _finish_hunting 0 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 1 monclient: found mon.noname-c 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.119+0000 7f94fd665640 1 -- 172.21.15.177:0/2632579775 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f806f030 con 0x7f94f809dc10 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94fd665640 1 -- 172.21.15.177:0/2632579775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94f809dc10 msgr2=0x7f94f809d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94fd665640 1 --2- 172.21.15.177:0/2632579775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94f809dc10 0x7f94f809d560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f94f80a3ab0 tx=0x7f94f40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=587e620240b2c93 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94effff640 1 -- 172.21.15.177:0/2632579775 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f402ae80 con 0x7f94f809dc10 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94effff640 10 monclient: dump: 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.135 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.136 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.136 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.136 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.136 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.136 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.136 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.136 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94effff640 10 monclient: _finish_auth 0 2024-09-15T03:58:29.136 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:29.136 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94effff640 1 -- 172.21.15.177:0/2632579775 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f4032440 con 0x7f94f809dc10 2024-09-15T03:58:29.136 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94ff8c9640 5 monclient: authenticate success, global_id 4108 2024-09-15T03:58:29.136 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94ff8c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:29.136 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94effff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:29.136 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94ff8c9640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:29.136 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94ff8c9640 4 set_mon_vals no callback set 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94effff640 1 -- 172.21.15.177:0/2632579775 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f40327c0 con 0x7f94f809dc10 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94effff640 10 monclient: dump: 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.137 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94ff8c9640 10 monclient: shutdown 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94ff8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.122+0000 7f94ff8c9640 1 -- 172.21.15.177:0/2632579775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94f809dc10 msgr2=0x7f94f809d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.123+0000 7f94ff8c9640 1 --2- 172.21.15.177:0/2632579775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94f809dc10 0x7f94f809d560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f94f80a3ab0 tx=0x7f94f40091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.123+0000 7f94ff8c9640 1 -- 172.21.15.177:0/2632579775 shutdown_connections 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.123+0000 7f94ff8c9640 1 --2- 172.21.15.177:0/2632579775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94f81346c0 0x7f94f8136aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.123+0000 7f94ff8c9640 1 --2- 172.21.15.177:0/2632579775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94f809dc10 0x7f94f809d560 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.123+0000 7f94ff8c9640 1 -- 172.21.15.177:0/2632579775 >> 172.21.15.177:0/2632579775 conn(0x7f94f8063e10 msgr2=0x7f94f80a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.123+0000 7f94ff8c9640 1 -- 172.21.15.177:0/2632579775 shutdown_connections 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.123+0000 7f94ff8c9640 1 -- 172.21.15.177:0/2632579775 wait complete. 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.123+0000 7f94ff8c9640 10 monclient: build_initial_monmap 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.123+0000 7f94ff8c9640 10 monclient: monmap: 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-09-15T03:58:29.138 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-09-15T03:58:29.139 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:29.139 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.139 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:29.139 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:29.139 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:29.139 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.123+0000 7f94ff8c9640 1 Processor -- start 2024-09-15T03:58:29.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 1 -- start start 2024-09-15T03:58:29.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 10 monclient: init 2024-09-15T03:58:29.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:29.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:58:29.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94f809dc10 0x7f94f8099310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 10 monclient(hunting): picked mon.noname-a con 0x7f94f809dc10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:29.139 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94f809dfe0 0x7f94f8099850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 10 monclient(hunting): picked mon.noname-c con 0x7f94f809dfe0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:29.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94f81346c0 0x7f94f8099d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 10 monclient(hunting): picked mon.noname-b con 0x7f94f81346c0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:29.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f812f630 con 0x7f94f809dc10 2024-09-15T03:58:29.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f809baa0 con 0x7f94f81346c0 2024-09-15T03:58:29.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f809b450 con 0x7f94f809dfe0 2024-09-15T03:58:29.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:29.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94ff8c9640 10 monclient(hunting): authenticate will time out at 733.176689s 2024-09-15T03:58:29.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fde66640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94f81346c0 0x7f94f8099d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fde66640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94f81346c0 0x7f94f8099d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:56114/0 (socket says 172.21.15.177:56114) 2024-09-15T03:58:29.140 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fde66640 1 -- 172.21.15.177:0/1726073969 learned_addr learned my addr 172.21.15.177:0/1726073969 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fde66640 10 monclient(hunting): get_auth_request con 0x7f94f81346c0 auth_method 0 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fde66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fde66640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fde66640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fce64640 1 --2- 172.21.15.177:0/1726073969 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94f809dfe0 0x7f94f8099850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fce64640 10 monclient(hunting): get_auth_request con 0x7f94f809dfe0 auth_method 0 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fce64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fce64640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fce64640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fd665640 1 --2- 172.21.15.177:0/1726073969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94f809dc10 0x7f94f8099310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fde66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fde66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fde66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fd665640 10 monclient(hunting): get_auth_request con 0x7f94f809dc10 auth_method 0 2024-09-15T03:58:29.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fd665640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fd665640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fd665640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fce64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fce64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.124+0000 7f94fce64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.125+0000 7f94fd665640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.125+0000 7f94fd665640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.125+0000 7f94fd665640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.125+0000 7f94fde66640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-09-15T03:58:29.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.125+0000 7f94fde66640 1 -- 172.21.15.177:0/1726073969 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94f809dfe0 msgr2=0x7f94f8099850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.125+0000 7f94fde66640 1 --2- 172.21.15.177:0/1726073969 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94f809dfe0 0x7f94f8099850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.125+0000 7f94fde66640 1 -- 172.21.15.177:0/1726073969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94f809dc10 msgr2=0x7f94f8099310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.125+0000 7f94fde66640 1 --2- 172.21.15.177:0/1726073969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94f809dc10 0x7f94f8099310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.125+0000 7f94fde66640 10 monclient: _finish_hunting 0 2024-09-15T03:58:29.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.125+0000 7f94fde66640 1 monclient: found mon.noname-b 2024-09-15T03:58:29.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.125+0000 7f94fde66640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:29.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.125+0000 7f94fde66640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.125+0000 7f94fde66640 1 -- 172.21.15.177:0/1726073969 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f4009450 con 0x7f94f81346c0 2024-09-15T03:58:29.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.125+0000 7f94fce64640 1 --2- 172.21.15.177:0/1726073969 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94f809dfe0 0x7f94f8099850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:29.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94fde66640 1 -- 172.21.15.177:0/1726073969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94f81346c0 msgr2=0x7f94f8099d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:29.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94fde66640 1 --2- 172.21.15.177:0/1726073969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94f81346c0 0x7f94f8099d90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f94f0009ec0 tx=0x7f94f000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:29.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ed7fa640 1 -- 172.21.15.177:0/1726073969 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f0014480 con 0x7f94f81346c0 2024-09-15T03:58:29.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ed7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:29.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ed7fa640 10 monclient: dump: 2024-09-15T03:58:29.143 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.143 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.143 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.144 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.144 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.144 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.144 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.144 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.144 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.144 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.144 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ed7fa640 10 monclient: _finish_auth 0 2024-09-15T03:58:29.144 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:29.144 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ed7fa640 1 -- 172.21.15.177:0/1726073969 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f000f040 con 0x7f94f81346c0 2024-09-15T03:58:29.144 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ff8c9640 5 monclient: authenticate success, global_id 4111 2024-09-15T03:58:29.144 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:29.144 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ed7fa640 1 -- 172.21.15.177:0/1726073969 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f0014810 con 0x7f94f81346c0 2024-09-15T03:58:29.144 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:29.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ed7fa640 10 monclient: dump: 2024-09-15T03:58:29.145 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.145 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.145 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.145 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.145 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.145 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.145 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.145 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.145 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.145 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ff8c9640 10 monclient: _renew_subs 2024-09-15T03:58:29.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ff8c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ef7fe640 4 set_mon_vals no callback set 2024-09-15T03:58:29.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ff8c9640 1 -- 172.21.15.177:0/1726073969 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f809ce10 con 0x7f94f81346c0 2024-09-15T03:58:29.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ff8c9640 10 monclient: _renew_subs 2024-09-15T03:58:29.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ff8c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ff8c9640 1 -- 172.21.15.177:0/1726073969 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f809af50 con 0x7f94f81346c0 2024-09-15T03:58:29.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ed7fa640 1 -- 172.21.15.177:0/1726073969 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f94f001d400 con 0x7f94f81346c0 2024-09-15T03:58:29.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.127+0000 7f94ed7fa640 1 -- 172.21.15.177:0/1726073969 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f94f0013730 con 0x7f94f81346c0 2024-09-15T03:58:29.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.128+0000 7f94ff8c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:29.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.128+0000 7f94ff8c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:29.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.128+0000 7f94ff8c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.128+0000 7f94ff8c9640 1 -- 172.21.15.177:0/1726073969 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f8070950 con 0x7f94f81346c0 2024-09-15T03:58:29.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.130+0000 7f94ed7fa640 1 -- 172.21.15.177:0/1726073969 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f94f003b7a0 con 0x7f94f81346c0 2024-09-15T03:58:29.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.130+0000 7f94ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:29.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.130+0000 7f94ed7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:29.198 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.197+0000 7f94ff8c9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "105a3b3d-e410-47fb-ac54-6a5d5da4a46f", "id": 0}] 2024-09-15T03:58:29.199 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.197+0000 7f94ff8c9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "105a3b3d-e410-47fb-ac54-6a5d5da4a46f", "id": 0}] 2024-09-15T03:58:29.199 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.197+0000 7f94ff8c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.199 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.197+0000 7f94ff8c9640 1 -- 172.21.15.177:0/1726073969 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "105a3b3d-e410-47fb-ac54-6a5d5da4a46f", "id": 0} v 0) v1 -- 0x7f94f806a450 con 0x7f94f81346c0 2024-09-15T03:58:29.202 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.201+0000 7f94ed7fa640 1 -- 172.21.15.177:0/1726073969 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "105a3b3d-e410-47fb-ac54-6a5d5da4a46f", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f94f002a070 con 0x7f94f81346c0 2024-09-15T03:58:29.202 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.201+0000 7f94ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "105a3b3d-e410-47fb-ac54-6a5d5da4a46f", "id": 0}] 2024-09-15T03:58:29.202 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.201+0000 7f94ed7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:29.203 INFO:teuthology.orchestra.run.smithi177.stdout:0 2024-09-15T03:58:29.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.202+0000 7f94ff8c9640 10 monclient: shutdown 2024-09-15T03:58:29.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.202+0000 7f94ff8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:29.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.202+0000 7f94ff8c9640 1 -- 172.21.15.177:0/1726073969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94f81346c0 msgr2=0x7f94f8099d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.202+0000 7f94ff8c9640 1 --2- 172.21.15.177:0/1726073969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94f81346c0 0x7f94f8099d90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f94f0009ec0 tx=0x7f94f000b040 comp rx=0 tx=0).stop 2024-09-15T03:58:29.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.202+0000 7f94ff8c9640 1 -- 172.21.15.177:0/1726073969 shutdown_connections 2024-09-15T03:58:29.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.202+0000 7f94ff8c9640 1 --2- 172.21.15.177:0/1726073969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f94f81346c0 0x7f94f8099d90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.202+0000 7f94ff8c9640 1 --2- 172.21.15.177:0/1726073969 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f94f809dfe0 0x7f94f8099850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.202+0000 7f94ff8c9640 1 --2- 172.21.15.177:0/1726073969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f94f809dc10 0x7f94f8099310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.202+0000 7f94ff8c9640 1 -- 172.21.15.177:0/1726073969 >> 172.21.15.177:0/1726073969 conn(0x7f94f8063e10 msgr2=0x7f94f80a0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:29.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.202+0000 7f94ff8c9640 1 -- 172.21.15.177:0/1726073969 shutdown_connections 2024-09-15T03:58:29.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.202+0000 7f94ff8c9640 1 -- 172.21.15.177:0/1726073969 wait complete. 2024-09-15T03:58:29.213 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 77f0f205-6c64-495a-8edd-043de180efa9 1 2024-09-15T03:58:29.305 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.304+0000 7f19e22af640 10 monclient: get_monmap_and_config 2024-09-15T03:58:29.306 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.304+0000 7f19e22af640 10 monclient: build_initial_monmap 2024-09-15T03:58:29.306 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.304+0000 7f19e22af640 10 monclient: monmap: 2024-09-15T03:58:29.306 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-09-15T03:58:29.306 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.306 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:29.306003+0000 2024-09-15T03:58:29.306 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:29.306003+0000 2024-09-15T03:58:29.306 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:29.306 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.306 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:29.306 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:29.306 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:29.306 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:29.307 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:29.307 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.309 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 1 Processor -- start 2024-09-15T03:58:29.309 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 1 -- start start 2024-09-15T03:58:29.309 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 10 monclient: init 2024-09-15T03:58:29.309 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:29.309 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-15T03:58:29.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f19dc0a17d0 0x7f19dc09f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 10 monclient(hunting): picked mon.noname-a con 0x7f19dc0a17d0 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:29.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f19dc09e7f0 0x7f19dc09ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 10 monclient(hunting): picked mon.noname-c con 0x7f19dc09e7f0 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:29.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f19dc131710 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:29.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc0a09e0 con 0x7f19dc131710 2024-09-15T03:58:29.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19dc0a0780 con 0x7f19dc0a17d0 2024-09-15T03:58:29.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19dc0a1ba0 con 0x7f19dc09e7f0 2024-09-15T03:58:29.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:29.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19e22af640 10 monclient(hunting): authenticate will time out at 733.357748s 2024-09-15T03:58:29.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19db7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f19dc09e7f0 0x7f19dc09ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.310 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19db7fe640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f19dc09e7f0 0x7f19dc09ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:56130/0 (socket says 172.21.15.177:56130) 2024-09-15T03:58:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19db7fe640 1 -- 172.21.15.177:0/2766807583 learned_addr learned my addr 172.21.15.177:0/2766807583 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-09-15T03:58:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19db7fe640 10 monclient(hunting): get_auth_request con 0x7f19dc09e7f0 auth_method 0 2024-09-15T03:58:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19dbfff640 1 --2- 172.21.15.177:0/2766807583 >> v2:172.21.15.110:3300/0 conn(0x7f19dc0a17d0 0x7f19dc09f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.305+0000 7f19db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 10 monclient(hunting): get_auth_request con 0x7f19dc0a17d0 auth_method 0 2024-09-15T03:58:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.311 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 10 monclient(hunting): handle_auth_done global_id 4110 payload 274 2024-09-15T03:58:29.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 1 -- 172.21.15.177:0/2766807583 >> v2:172.21.15.177:3300/0 conn(0x7f19dc09e7f0 msgr2=0x7f19dc09ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:29.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 1 --2- 172.21.15.177:0/2766807583 >> v2:172.21.15.177:3300/0 conn(0x7f19dc09e7f0 0x7f19dc09ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 1 -- 172.21.15.177:0/2766807583 >> v1:172.21.15.177:6789/0 conn(0x7f19dc131710 legacy=0x7f19dc131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:29.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:29.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 1 monclient: found mon.noname-a 2024-09-15T03:58:29.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:29.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:29.312 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.306+0000 7f19dbfff640 1 -- 172.21.15.177:0/2766807583 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc09ca20 con 0x7f19dc0a17d0 2024-09-15T03:58:29.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19dbfff640 1 -- 172.21.15.177:0/2766807583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19dc0a17d0 msgr2=0x7f19dc09f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:29.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19dbfff640 1 --2- 172.21.15.177:0/2766807583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19dc0a17d0 0x7f19dc09f840 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f19cc009850 tx=0x7f19cc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd90365d97afb5c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:29.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19daffd640 1 -- 172.21.15.177:0/2766807583 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc032440 con 0x7f19dc0a17d0 2024-09-15T03:58:29.319 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.320 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19daffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:29.320 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19daffd640 10 monclient: dump: 2024-09-15T03:58:29.320 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.320 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.320 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.320 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.320 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.320 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.320 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.320 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.320 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.320 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.320 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19daffd640 10 monclient: _finish_auth 0 2024-09-15T03:58:29.320 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:29.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19daffd640 1 -- 172.21.15.177:0/2766807583 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19cc0325a0 con 0x7f19dc0a17d0 2024-09-15T03:58:29.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19e22af640 5 monclient: authenticate success, global_id 4110 2024-09-15T03:58:29.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19e22af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:29.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19daffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:29.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19e22af640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:29.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19daffd640 1 -- 172.21.15.177:0/2766807583 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc032940 con 0x7f19dc0a17d0 2024-09-15T03:58:29.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19e22af640 4 set_mon_vals no callback set 2024-09-15T03:58:29.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:29.321 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19daffd640 10 monclient: dump: 2024-09-15T03:58:29.321 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.321 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.321 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.321 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.322 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.322 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.322 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.322 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.322 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.322 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.322 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19e22af640 10 monclient: shutdown 2024-09-15T03:58:29.322 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19e22af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:29.322 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19e22af640 1 -- 172.21.15.177:0/2766807583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19dc0a17d0 msgr2=0x7f19dc09f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.322 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19e22af640 1 --2- 172.21.15.177:0/2766807583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19dc0a17d0 0x7f19dc09f840 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f19cc009850 tx=0x7f19cc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.322 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19e22af640 1 -- 172.21.15.177:0/2766807583 shutdown_connections 2024-09-15T03:58:29.322 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19e22af640 1 --2- 172.21.15.177:0/2766807583 >> v2:172.21.15.177:3300/0 conn(0x7f19dc09e7f0 0x7f19dc09ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.322 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19e22af640 1 --2- 172.21.15.177:0/2766807583 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19dc0a17d0 0x7f19dc09f840 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.322 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19e22af640 1 -- 172.21.15.177:0/2766807583 >> 172.21.15.177:0/2766807583 conn(0x7f19dc0631f0 msgr2=0x7f19dc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:29.323 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.309+0000 7f19e22af640 1 -- 172.21.15.177:0/2766807583 shutdown_connections 2024-09-15T03:58:29.323 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.310+0000 7f19e22af640 1 -- 172.21.15.177:0/2766807583 wait complete. 2024-09-15T03:58:29.323 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.310+0000 7f19e22af640 10 monclient: build_initial_monmap 2024-09-15T03:58:29.323 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.310+0000 7f19e22af640 10 monclient: monmap: 2024-09-15T03:58:29.323 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-09-15T03:58:29.323 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:29.323 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-09-15T03:58:29.324 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-09-15T03:58:29.324 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:29.324 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.324 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:29.324 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:29.324 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:29.324 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.310+0000 7f19e22af640 1 Processor -- start 2024-09-15T03:58:29.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.310+0000 7f19e22af640 1 -- start start 2024-09-15T03:58:29.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.310+0000 7f19e22af640 10 monclient: init 2024-09-15T03:58:29.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:29.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:58:29.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19dc09e7f0 0x7f19dc1305a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.324 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 10 monclient(hunting): picked mon.noname-a con 0x7f19dc09e7f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:29.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19dc0a17d0 0x7f19dc11b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 10 monclient(hunting): picked mon.noname-b con 0x7f19dc0a17d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:29.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f19dc131710 0x7f19dc11b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 10 monclient(hunting): picked mon.noname-c con 0x7f19dc131710 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:29.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc11e9b0 con 0x7f19dc09e7f0 2024-09-15T03:58:29.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc11e6d0 con 0x7f19dc0a17d0 2024-09-15T03:58:29.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc11ec90 con 0x7f19dc131710 2024-09-15T03:58:29.329 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:29.329 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e22af640 10 monclient(hunting): authenticate will time out at 733.363439s 2024-09-15T03:58:29.329 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19dc0a17d0 0x7f19dc11b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.329 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19dbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19dc09e7f0 0x7f19dc1305a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.329 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19dc0a17d0 0x7f19dc11b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:56140/0 (socket says 172.21.15.177:56140) 2024-09-15T03:58:29.330 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 1 -- 172.21.15.177:0/535470053 learned_addr learned my addr 172.21.15.177:0/535470053 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-09-15T03:58:29.330 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 10 monclient(hunting): get_auth_request con 0x7f19dc0a17d0 auth_method 0 2024-09-15T03:58:29.330 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.330 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.330 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.330 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19dbfff640 10 monclient(hunting): get_auth_request con 0x7f19dc09e7f0 auth_method 0 2024-09-15T03:58:29.330 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.330 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.330 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.330 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.330 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.330 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e084c640 1 --2- 172.21.15.177:0/535470053 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f19dc131710 0x7f19dc11b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.330 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.330 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e084c640 10 monclient(hunting): get_auth_request con 0x7f19dc131710 auth_method 0 2024-09-15T03:58:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e084c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e084c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e084c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e084c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e084c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19e084c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-09-15T03:58:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 1 -- 172.21.15.177:0/535470053 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f19dc131710 msgr2=0x7f19dc11b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 1 --2- 172.21.15.177:0/535470053 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f19dc131710 0x7f19dc11b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 1 -- 172.21.15.177:0/535470053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19dc09e7f0 msgr2=0x7f19dc1305a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 1 --2- 172.21.15.177:0/535470053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19dc09e7f0 0x7f19dc1305a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.332 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:58:29.332 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 1 monclient: found mon.noname-b 2024-09-15T03:58:29.332 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:29.332 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.332 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19db7fe640 1 -- 172.21.15.177:0/535470053 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc009390 con 0x7f19dc0a17d0 2024-09-15T03:58:29.332 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.311+0000 7f19dbfff640 1 --2- 172.21.15.177:0/535470053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19dc09e7f0 0x7f19dc1305a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:29.332 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19db7fe640 1 -- 172.21.15.177:0/535470053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19dc0a17d0 msgr2=0x7f19dc11b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:29.332 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19db7fe640 1 --2- 172.21.15.177:0/535470053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19dc0a17d0 0x7f19dc11b3d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f19d000ea20 tx=0x7f19d00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:29.332 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19c3fff640 1 -- 172.21.15.177:0/535470053 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d000cb50 con 0x7f19dc0a17d0 2024-09-15T03:58:29.332 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.332 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19c3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:29.332 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19c3fff640 10 monclient: dump: 2024-09-15T03:58:29.332 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.333 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.333 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.333 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.333 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.333 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.333 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.333 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.333 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.333 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.333 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19c3fff640 10 monclient: _finish_auth 0 2024-09-15T03:58:29.333 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:29.334 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19c3fff640 1 -- 172.21.15.177:0/535470053 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d000ccb0 con 0x7f19dc0a17d0 2024-09-15T03:58:29.334 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19c3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:29.334 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19c3fff640 1 -- 172.21.15.177:0/535470053 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d0013470 con 0x7f19dc0a17d0 2024-09-15T03:58:29.334 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19e22af640 5 monclient: authenticate success, global_id 4117 2024-09-15T03:58:29.334 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19e22af640 10 monclient: _renew_subs 2024-09-15T03:58:29.334 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19e22af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.334 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19e22af640 1 -- 172.21.15.177:0/535470053 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc11cba0 con 0x7f19dc0a17d0 2024-09-15T03:58:29.334 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.334 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:29.334 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19c3fff640 10 monclient: dump: 2024-09-15T03:58:29.334 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.334 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.334 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19da7fc640 4 set_mon_vals no callback set 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19e22af640 10 monclient: _renew_subs 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19e22af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19e22af640 1 -- 172.21.15.177:0/535470053 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc141b60 con 0x7f19dc0a17d0 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19c3fff640 1 -- 172.21.15.177:0/535470053 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f19d0013860 con 0x7f19dc0a17d0 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.314+0000 7f19c3fff640 1 -- 172.21.15.177:0/535470053 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f19d001c9f0 con 0x7f19dc0a17d0 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.315+0000 7f19e22af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:29.335 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.315+0000 7f19e22af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:29.336 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.315+0000 7f19e22af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.336 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.315+0000 7f19e22af640 1 -- 172.21.15.177:0/535470053 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc131340 con 0x7f19dc0a17d0 2024-09-15T03:58:29.336 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.317+0000 7f19c3fff640 1 -- 172.21.15.177:0/535470053 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f19d0021060 con 0x7f19dc0a17d0 2024-09-15T03:58:29.336 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.317+0000 7f19c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:29.336 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.317+0000 7f19c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:29.376 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:29.375+0000 7fab3c215ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T03:58:29.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.383+0000 7f19e22af640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "77f0f205-6c64-495a-8edd-043de180efa9", "id": 1}] 2024-09-15T03:58:29.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.383+0000 7f19e22af640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "77f0f205-6c64-495a-8edd-043de180efa9", "id": 1}] 2024-09-15T03:58:29.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.383+0000 7f19e22af640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.385 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.383+0000 7f19e22af640 1 -- 172.21.15.177:0/535470053 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "77f0f205-6c64-495a-8edd-043de180efa9", "id": 1} v 0) v1 -- 0x7f19dc070950 con 0x7f19dc0a17d0 2024-09-15T03:58:29.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.387+0000 7f19c3fff640 1 -- 172.21.15.177:0/535470053 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "77f0f205-6c64-495a-8edd-043de180efa9", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f19d0017070 con 0x7f19dc0a17d0 2024-09-15T03:58:29.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.387+0000 7f19c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "77f0f205-6c64-495a-8edd-043de180efa9", "id": 1}] 2024-09-15T03:58:29.389 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.387+0000 7f19c3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:29.389 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-09-15T03:58:29.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.388+0000 7f19e22af640 10 monclient: shutdown 2024-09-15T03:58:29.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.388+0000 7f19e22af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:29.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.388+0000 7f19e22af640 1 -- 172.21.15.177:0/535470053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19dc0a17d0 msgr2=0x7f19dc11b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.388+0000 7f19e22af640 1 --2- 172.21.15.177:0/535470053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19dc0a17d0 0x7f19dc11b3d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f19d000ea20 tx=0x7f19d00091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.388+0000 7f19e22af640 1 -- 172.21.15.177:0/535470053 shutdown_connections 2024-09-15T03:58:29.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.388+0000 7f19e22af640 1 --2- 172.21.15.177:0/535470053 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f19dc131710 0x7f19dc11b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.388+0000 7f19e22af640 1 --2- 172.21.15.177:0/535470053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f19dc0a17d0 0x7f19dc11b3d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.390 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.388+0000 7f19e22af640 1 --2- 172.21.15.177:0/535470053 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f19dc09e7f0 0x7f19dc1305a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.391 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.388+0000 7f19e22af640 1 -- 172.21.15.177:0/535470053 >> 172.21.15.177:0/535470053 conn(0x7f19dc0631f0 msgr2=0x7f19dc09f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:29.391 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.388+0000 7f19e22af640 1 -- 172.21.15.177:0/535470053 shutdown_connections 2024-09-15T03:58:29.391 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.388+0000 7f19e22af640 1 -- 172.21.15.177:0/535470053 wait complete. 2024-09-15T03:58:29.400 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new e8a42031-303e-4ad0-9896-e5b84cc95cfb 2 2024-09-15T03:58:29.433 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:29.432+0000 7fab3c215ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T03:58:29.492 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.491+0000 7fd69731d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:29.493 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.491+0000 7fd69731d640 10 monclient: build_initial_monmap 2024-09-15T03:58:29.493 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.491+0000 7fd69731d640 10 monclient: monmap: 2024-09-15T03:58:29.493 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-09-15T03:58:29.493 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.493 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:29.492622+0000 2024-09-15T03:58:29.493 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:29.492622+0000 2024-09-15T03:58:29.493 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:29.493 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.493 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:29.493 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:29.493 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:29.493 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:29.494 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:29.494 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.494 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.491+0000 7fd69731d640 1 Processor -- start 2024-09-15T03:58:29.494 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.491+0000 7fd69731d640 1 -- start start 2024-09-15T03:58:29.494 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.491+0000 7fd69731d640 10 monclient: init 2024-09-15T03:58:29.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd69731d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:29.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd69731d640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T03:58:29.498 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd69731d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6900a17d0 0x7fd69009f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd69731d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6900a17d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:29.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd69731d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd69009e7f0 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:29.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd69731d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd690136a40 0x7fd690138e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd69731d640 10 monclient(hunting): picked mon.noname-a con 0x7fd690136a40 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:29.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd69731d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6900a09e0 con 0x7fd69009e7f0 2024-09-15T03:58:29.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd69731d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd69731d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6900a0760 con 0x7fd690136a40 2024-09-15T03:58:29.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd69731d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd69731d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6900a1ba0 con 0x7fd6900a17d0 2024-09-15T03:58:29.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd69731d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:29.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd69731d640 10 monclient(hunting): authenticate will time out at 733.544366s 2024-09-15T03:58:29.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6958ba640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd690136a40 0x7fd690138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.499 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6900a17d0 0x7fd69009f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.500 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6958ba640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd690136a40 0x7fd690138e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.177:55582/0 (socket says 172.21.15.177:55582) 2024-09-15T03:58:29.500 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6948b8640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd69009e7f0 0x7fd69009ec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.177:56660/0 (socket says 172.21.15.177:56660) 2024-09-15T03:58:29.500 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6900a17d0 0x7fd69009f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.177:35112/0 (socket says 172.21.15.177:35112) 2024-09-15T03:58:29.500 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6958ba640 1 -- 172.21.15.177:0/774650282 learned_addr learned my addr 172.21.15.177:0/774650282 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-09-15T03:58:29.500 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6958ba640 10 monclient(hunting): get_auth_request con 0x7fd690136a40 auth_method 0 2024-09-15T03:58:29.500 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6958ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.500 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6958ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.500 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6958ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.500 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 10 monclient(hunting): get_auth_request con 0x7fd6900a17d0 auth_method 0 2024-09-15T03:58:29.500 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.500 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.500 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.500 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6958ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6958ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6958ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2024-09-15T03:58:29.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 1 -- 172.21.15.177:0/774650282 >> v2:172.21.15.110:3300/0 conn(0x7fd690136a40 msgr2=0x7fd690138e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:29.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 1 --2- 172.21.15.177:0/774650282 >> v2:172.21.15.110:3300/0 conn(0x7fd690136a40 0x7fd690138e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 1 -- 172.21.15.177:0/774650282 >> v1:172.21.15.110:6789/0 conn(0x7fd69009e7f0 legacy=0x7fd69009ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:29.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 10 monclient: _finish_hunting 0 2024-09-15T03:58:29.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 1 monclient: found mon.noname-b 2024-09-15T03:58:29.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:29.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:29.501 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6950b9640 1 -- 172.21.15.177:0/774650282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69009ca20 con 0x7fd6900a17d0 2024-09-15T03:58:29.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.492+0000 7fd6958ba640 1 --2- 172.21.15.177:0/774650282 >> v2:172.21.15.110:3300/0 conn(0x7fd690136a40 0x7fd690138e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:29.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd6950b9640 1 -- 172.21.15.177:0/774650282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6900a17d0 msgr2=0x7fd69009f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:29.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd6950b9640 1 --2- 172.21.15.177:0/774650282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6900a17d0 0x7fd69009f840 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd68c009850 tx=0x7fd68c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72cb82315274b4ce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:29.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd687fff640 1 -- 172.21.15.177:0/774650282 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c02ae80 con 0x7fd6900a17d0 2024-09-15T03:58:29.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd687fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd687fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:29.502 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd687fff640 10 monclient: dump: 2024-09-15T03:58:29.502 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.502 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.502 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.502 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.502 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.502 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.502 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.503 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.503 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.503 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd687fff640 10 monclient: _finish_auth 0 2024-09-15T03:58:29.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd687fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:29.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd69731d640 5 monclient: authenticate success, global_id 4112 2024-09-15T03:58:29.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd69731d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:29.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd687fff640 1 -- 172.21.15.177:0/774650282 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68c032440 con 0x7fd6900a17d0 2024-09-15T03:58:29.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd687fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:29.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd69731d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:29.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd687fff640 1 -- 172.21.15.177:0/774650282 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c0327c0 con 0x7fd6900a17d0 2024-09-15T03:58:29.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd69731d640 4 set_mon_vals no callback set 2024-09-15T03:58:29.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd687fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.503 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd687fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:29.504 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.495+0000 7fd687fff640 10 monclient: dump: 2024-09-15T03:58:29.504 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.504 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.504 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.504 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.504 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.504 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.504 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.504 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.504 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.504 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.504 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.496+0000 7fd69731d640 10 monclient: shutdown 2024-09-15T03:58:29.504 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.496+0000 7fd69731d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:29.505 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.496+0000 7fd69731d640 1 -- 172.21.15.177:0/774650282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6900a17d0 msgr2=0x7fd69009f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.505 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.496+0000 7fd69731d640 1 --2- 172.21.15.177:0/774650282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6900a17d0 0x7fd69009f840 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd68c009850 tx=0x7fd68c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.505 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.496+0000 7fd69731d640 1 -- 172.21.15.177:0/774650282 shutdown_connections 2024-09-15T03:58:29.505 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.496+0000 7fd69731d640 1 --2- 172.21.15.177:0/774650282 >> v2:172.21.15.110:3300/0 conn(0x7fd690136a40 0x7fd690138e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.505 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.496+0000 7fd69731d640 1 --2- 172.21.15.177:0/774650282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6900a17d0 0x7fd69009f840 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.505 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.496+0000 7fd69731d640 1 -- 172.21.15.177:0/774650282 >> 172.21.15.177:0/774650282 conn(0x7fd6900631f0 msgr2=0x7fd6900635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:29.505 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.496+0000 7fd69731d640 1 -- 172.21.15.177:0/774650282 shutdown_connections 2024-09-15T03:58:29.505 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.496+0000 7fd69731d640 1 -- 172.21.15.177:0/774650282 wait complete. 2024-09-15T03:58:29.505 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.497+0000 7fd69731d640 10 monclient: build_initial_monmap 2024-09-15T03:58:29.505 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.497+0000 7fd69731d640 10 monclient: monmap: 2024-09-15T03:58:29.505 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-09-15T03:58:29.505 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:29.505 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-09-15T03:58:29.505 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-09-15T03:58:29.506 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:29.506 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.506 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:29.506 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:29.506 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:29.506 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.506 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.497+0000 7fd69731d640 1 Processor -- start 2024-09-15T03:58:29.506 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.497+0000 7fd69731d640 1 -- start start 2024-09-15T03:58:29.506 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.497+0000 7fd69731d640 10 monclient: init 2024-09-15T03:58:29.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:29.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:58:29.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd69009e7f0 0x7fd690140f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 10 monclient(hunting): picked mon.noname-a con 0x7fd69009e7f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:29.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6900a17d0 0x7fd6901336f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6900a17d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:29.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd690136a40 0x7fd69011c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 10 monclient(hunting): picked mon.noname-c con 0x7fd690136a40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:29.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69011e290 con 0x7fd69009e7f0 2024-09-15T03:58:29.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69009c250 con 0x7fd6900a17d0 2024-09-15T03:58:29.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69011fae0 con 0x7fd690136a40 2024-09-15T03:58:29.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:29.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd69731d640 10 monclient(hunting): authenticate will time out at 733.550141s 2024-09-15T03:58:29.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6948b8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6900a17d0 0x7fd6901336f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6948b8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6900a17d0 0x7fd6901336f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:56148/0 (socket says 172.21.15.177:56148) 2024-09-15T03:58:29.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6948b8640 1 -- 172.21.15.177:0/2450400698 learned_addr learned my addr 172.21.15.177:0/2450400698 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-09-15T03:58:29.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6948b8640 10 monclient(hunting): get_auth_request con 0x7fd6900a17d0 auth_method 0 2024-09-15T03:58:29.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6948b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6948b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.512 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6948b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 1 --2- 172.21.15.177:0/2450400698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd69009e7f0 0x7fd690140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 10 monclient(hunting): get_auth_request con 0x7fd69009e7f0 auth_method 0 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6958ba640 1 --2- 172.21.15.177:0/2450400698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd690136a40 0x7fd69011c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6958ba640 10 monclient(hunting): get_auth_request con 0x7fd690136a40 auth_method 0 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6958ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6958ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6958ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6948b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6948b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6948b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.513 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6958ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6958ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6958ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 10 monclient(hunting): handle_auth_done global_id 4119 payload 824 2024-09-15T03:58:29.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 1 -- 172.21.15.177:0/2450400698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd690136a40 msgr2=0x7fd69011c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 1 --2- 172.21.15.177:0/2450400698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd690136a40 0x7fd69011c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 1 -- 172.21.15.177:0/2450400698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6900a17d0 msgr2=0x7fd6901336f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 1 --2- 172.21.15.177:0/2450400698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6900a17d0 0x7fd6901336f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 10 monclient: _finish_hunting 0 2024-09-15T03:58:29.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 1 monclient: found mon.noname-a 2024-09-15T03:58:29.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:29.514 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:29.515 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6950b9640 1 -- 172.21.15.177:0/2450400698 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd688009ae0 con 0x7fd69009e7f0 2024-09-15T03:58:29.515 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6948b8640 1 --2- 172.21.15.177:0/2450400698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6900a17d0 0x7fd6901336f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:29.515 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.498+0000 7fd6958ba640 1 --2- 172.21.15.177:0/2450400698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd690136a40 0x7fd69011c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:29.515 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.500+0000 7fd6950b9640 1 -- 172.21.15.177:0/2450400698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd69009e7f0 msgr2=0x7fd690140f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:29.515 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.500+0000 7fd6950b9640 1 --2- 172.21.15.177:0/2450400698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd69009e7f0 0x7fd690140f20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd68c034000 tx=0x7fd68c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:29.515 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6857fa640 1 -- 172.21.15.177:0/2450400698 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c031c70 con 0x7fd69009e7f0 2024-09-15T03:58:29.515 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.515 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6857fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:29.515 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6857fa640 10 monclient: dump: 2024-09-15T03:58:29.515 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.515 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.515 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.515 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.515 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.516 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.516 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.516 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.516 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.516 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6857fa640 10 monclient: _finish_auth 0 2024-09-15T03:58:29.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:29.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd69731d640 5 monclient: authenticate success, global_id 4119 2024-09-15T03:58:29.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd69731d640 10 monclient: _renew_subs 2024-09-15T03:58:29.516 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd69731d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:29.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd69731d640 1 -- 172.21.15.177:0/2450400698 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68c009390 con 0x7fd69009e7f0 2024-09-15T03:58:29.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd69731d640 10 monclient: _renew_subs 2024-09-15T03:58:29.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd69731d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:29.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd69731d640 1 -- 172.21.15.177:0/2450400698 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69011dcd0 con 0x7fd69009e7f0 2024-09-15T03:58:29.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6857fa640 1 -- 172.21.15.177:0/2450400698 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68c031e10 con 0x7fd69009e7f0 2024-09-15T03:58:29.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6857fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:29.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6857fa640 1 -- 172.21.15.177:0/2450400698 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c03c650 con 0x7fd69009e7f0 2024-09-15T03:58:29.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:29.517 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6857fa640 10 monclient: dump: 2024-09-15T03:58:29.517 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.517 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.517 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6877fe640 4 set_mon_vals no callback set 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6857fa640 1 -- 172.21.15.177:0/2450400698 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fd68c03ca00 con 0x7fd69009e7f0 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.501+0000 7fd6857fa640 1 -- 172.21.15.177:0/2450400698 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7fd68c04d7d0 con 0x7fd69009e7f0 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.502+0000 7fd69731d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.502+0000 7fd69731d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.502+0000 7fd69731d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.502+0000 7fd69731d640 1 -- 172.21.15.177:0/2450400698 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd670005010 con 0x7fd69009e7f0 2024-09-15T03:58:29.518 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.504+0000 7fd6857fa640 1 -- 172.21.15.177:0/2450400698 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7fd68c049060 con 0x7fd69009e7f0 2024-09-15T03:58:29.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.504+0000 7fd6857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:29.519 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.504+0000 7fd6857fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:29.569 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:29.567+0000 7fab3c215ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T03:58:29.571 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.569+0000 7fd69731d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e8a42031-303e-4ad0-9896-e5b84cc95cfb", "id": 2}] 2024-09-15T03:58:29.571 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.569+0000 7fd69731d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e8a42031-303e-4ad0-9896-e5b84cc95cfb", "id": 2}] 2024-09-15T03:58:29.571 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.569+0000 7fd69731d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:29.571 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.569+0000 7fd69731d640 1 -- 172.21.15.177:0/2450400698 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e8a42031-303e-4ad0-9896-e5b84cc95cfb", "id": 2} v 0) v1 -- 0x7fd670005600 con 0x7fd69009e7f0 2024-09-15T03:58:29.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.572+0000 7fd6857fa640 1 -- 172.21.15.177:0/2450400698 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e8a42031-303e-4ad0-9896-e5b84cc95cfb", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd68c037070 con 0x7fd69009e7f0 2024-09-15T03:58:29.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.572+0000 7fd6857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e8a42031-303e-4ad0-9896-e5b84cc95cfb", "id": 2}] 2024-09-15T03:58:29.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.572+0000 7fd6857fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:29.574 INFO:teuthology.orchestra.run.smithi177.stdout:2 2024-09-15T03:58:29.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.573+0000 7fd69731d640 10 monclient: shutdown 2024-09-15T03:58:29.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.573+0000 7fd69731d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:29.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.573+0000 7fd69731d640 1 -- 172.21.15.177:0/2450400698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd69009e7f0 msgr2=0x7fd690140f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.573+0000 7fd69731d640 1 --2- 172.21.15.177:0/2450400698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd69009e7f0 0x7fd690140f20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd68c034000 tx=0x7fd68c032000 comp rx=0 tx=0).stop 2024-09-15T03:58:29.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.573+0000 7fd69731d640 1 -- 172.21.15.177:0/2450400698 shutdown_connections 2024-09-15T03:58:29.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.573+0000 7fd69731d640 1 --2- 172.21.15.177:0/2450400698 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd690136a40 0x7fd69011c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.573+0000 7fd69731d640 1 --2- 172.21.15.177:0/2450400698 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6900a17d0 0x7fd6901336f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.573+0000 7fd69731d640 1 --2- 172.21.15.177:0/2450400698 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd69009e7f0 0x7fd690140f20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.573+0000 7fd69731d640 1 -- 172.21.15.177:0/2450400698 >> 172.21.15.177:0/2450400698 conn(0x7fd6900631f0 msgr2=0x7fd690095ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:29.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.573+0000 7fd69731d640 1 -- 172.21.15.177:0/2450400698 shutdown_connections 2024-09-15T03:58:29.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.573+0000 7fd69731d640 1 -- 172.21.15.177:0/2450400698 wait complete. 2024-09-15T03:58:29.585 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new 0d65db82-4a0e-4f31-a10a-dcd71dcb269f 3 2024-09-15T03:58:29.720 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.718+0000 7fe31ff5d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:29.720 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.718+0000 7fe31ff5d640 10 monclient: build_initial_monmap 2024-09-15T03:58:29.720 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.718+0000 7fe31ff5d640 10 monclient: monmap: 2024-09-15T03:58:29.720 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-09-15T03:58:29.720 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.720 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:29.720450+0000 2024-09-15T03:58:29.721 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:29.720450+0000 2024-09-15T03:58:29.721 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:29.721 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.721 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:29.721 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:29.721 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:29.721 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:29.721 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:29.721 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.724 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.719+0000 7fe31ff5d640 1 Processor -- start 2024-09-15T03:58:29.724 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.719+0000 7fe31ff5d640 1 -- start start 2024-09-15T03:58:29.724 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.719+0000 7fe31ff5d640 10 monclient: init 2024-09-15T03:58:29.724 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:29.724 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-15T03:58:29.724 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe3180a17d0 0x7fe31809f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.724 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 10 monclient(hunting): picked mon.noname-a con 0x7fe3180a17d0 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:29.724 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe31809e7f0 0x7fe31809ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.724 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 10 monclient(hunting): picked mon.noname-c con 0x7fe31809e7f0 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:29.724 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe318131710 0x7fe318131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.724 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 10 monclient(hunting): picked mon.noname-b con 0x7fe318131710 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:29.724 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.724 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3180a0740 con 0x7fe3180a17d0 2024-09-15T03:58:29.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3180a09e0 con 0x7fe31809e7f0 2024-09-15T03:58:29.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3180a1ba0 con 0x7fe318131710 2024-09-15T03:58:29.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:29.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31ff5d640 10 monclient(hunting): authenticate will time out at 733.772144s 2024-09-15T03:58:29.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe31809e7f0 0x7fe31809ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe31809e7f0 0x7fe31809ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:56158/0 (socket says 172.21.15.177:56158) 2024-09-15T03:58:29.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 1 -- 172.21.15.177:0/1021312664 learned_addr learned my addr 172.21.15.177:0/1021312664 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-09-15T03:58:29.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 10 monclient(hunting): get_auth_request con 0x7fe31809e7f0 auth_method 0 2024-09-15T03:58:29.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31e4fa640 1 --2- 172.21.15.177:0/1021312664 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe318131710 0x7fe318131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31dcf9640 1 --2- 172.21.15.177:0/1021312664 >> v2:172.21.15.110:3300/0 conn(0x7fe3180a17d0 0x7fe31809f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31e4fa640 10 monclient(hunting): get_auth_request con 0x7fe318131710 auth_method 0 2024-09-15T03:58:29.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31e4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31e4fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31e4fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31dcf9640 10 monclient(hunting): get_auth_request con 0x7fe3180a17d0 auth_method 0 2024-09-15T03:58:29.727 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31dcf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.727 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31dcf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.727 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31dcf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.727 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31e4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.727 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31e4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31e4fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31dcf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31dcf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31dcf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2024-09-15T03:58:29.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 1 -- 172.21.15.177:0/1021312664 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe318131710 msgr2=0x7fe318131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:29.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 1 --2- 172.21.15.177:0/1021312664 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe318131710 0x7fe318131b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 1 -- 172.21.15.177:0/1021312664 >> v2:172.21.15.110:3300/0 conn(0x7fe3180a17d0 msgr2=0x7fe31809f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:29.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 1 --2- 172.21.15.177:0/1021312664 >> v2:172.21.15.110:3300/0 conn(0x7fe3180a17d0 0x7fe31809f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.728 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 10 monclient: _finish_hunting 0 2024-09-15T03:58:29.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 1 monclient: found mon.noname-c 2024-09-15T03:58:29.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:29.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.729 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.720+0000 7fe31d4f8640 1 -- 172.21.15.177:0/1021312664 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31809ca20 con 0x7fe31809e7f0 2024-09-15T03:58:29.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31d4f8640 1 -- 172.21.15.177:0/1021312664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe31809e7f0 msgr2=0x7fe31809ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:29.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31d4f8640 1 --2- 172.21.15.177:0/1021312664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe31809e7f0 0x7fe31809ec50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009910 tx=0x7fe30c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ec36aec64365c42 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:29.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ccf7640 1 -- 172.21.15.177:0/1021312664 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c02aea0 con 0x7fe31809e7f0 2024-09-15T03:58:29.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ccf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ccf7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:29.737 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ccf7640 10 monclient: dump: 2024-09-15T03:58:29.737 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.737 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.737 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ccf7640 10 monclient: _finish_auth 0 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ccf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ccf7640 1 -- 172.21.15.177:0/1021312664 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30c032440 con 0x7fe31809e7f0 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ff5d640 5 monclient: authenticate success, global_id 4123 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ff5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ccf7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ff5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:29.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ccf7640 1 -- 172.21.15.177:0/1021312664 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c0327e0 con 0x7fe31809e7f0 2024-09-15T03:58:29.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ff5d640 4 set_mon_vals no callback set 2024-09-15T03:58:29.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ccf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ccf7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:29.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.723+0000 7fe31ccf7640 10 monclient: dump: 2024-09-15T03:58:29.739 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.739 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.739 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.739 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.739 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.739 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.739 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.740 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.740 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.740 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.740 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 10 monclient: shutdown 2024-09-15T03:58:29.740 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:29.740 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 1 -- 172.21.15.177:0/1021312664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe31809e7f0 msgr2=0x7fe31809ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.740 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 1 --2- 172.21.15.177:0/1021312664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe31809e7f0 0x7fe31809ec50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009910 tx=0x7fe30c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.740 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 1 -- 172.21.15.177:0/1021312664 shutdown_connections 2024-09-15T03:58:29.740 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 1 --2- 172.21.15.177:0/1021312664 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe318131710 0x7fe318131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.740 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 1 --2- 172.21.15.177:0/1021312664 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe31809e7f0 0x7fe31809ec50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.740 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 1 --2- 172.21.15.177:0/1021312664 >> v2:172.21.15.110:3300/0 conn(0x7fe3180a17d0 0x7fe31809f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.740 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 1 -- 172.21.15.177:0/1021312664 >> 172.21.15.177:0/1021312664 conn(0x7fe3180631f0 msgr2=0x7fe3180635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 1 -- 172.21.15.177:0/1021312664 shutdown_connections 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 1 -- 172.21.15.177:0/1021312664 wait complete. 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 10 monclient: build_initial_monmap 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 10 monclient: monmap: 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 1 Processor -- start 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 1 -- start start 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.724+0000 7fe31ff5d640 10 monclient: init 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe31809e7f0 0x7fe318122ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 10 monclient(hunting): picked mon.noname-c con 0x7fe31809e7f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3180a17d0 0x7fe318141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 10 monclient(hunting): picked mon.noname-b con 0x7fe3180a17d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318131710 0x7fe318141750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 10 monclient(hunting): picked mon.noname-a con 0x7fe318131710 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe318121ee0 con 0x7fe318131710 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3180a40a0 con 0x7fe3180a17d0 2024-09-15T03:58:29.742 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31809c280 con 0x7fe31809e7f0 2024-09-15T03:58:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31ff5d640 10 monclient(hunting): authenticate will time out at 733.777402s 2024-09-15T03:58:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3180a17d0 0x7fe318141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3180a17d0 0x7fe318141210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:56168/0 (socket says 172.21.15.177:56168) 2024-09-15T03:58:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 1 -- 172.21.15.177:0/2645547755 learned_addr learned my addr 172.21.15.177:0/2645547755 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-09-15T03:58:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 10 monclient(hunting): get_auth_request con 0x7fe3180a17d0 auth_method 0 2024-09-15T03:58:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:29.743 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31dcf9640 1 --2- 172.21.15.177:0/2645547755 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe31809e7f0 0x7fe318122ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31e4fa640 1 --2- 172.21.15.177:0/2645547755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318131710 0x7fe318141750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:29.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31dcf9640 10 monclient(hunting): get_auth_request con 0x7fe31809e7f0 auth_method 0 2024-09-15T03:58:29.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31dcf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:29.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31dcf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:29.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31dcf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:29.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-09-15T03:58:29.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 1 -- 172.21.15.177:0/2645547755 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe31809e7f0 msgr2=0x7fe318122ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 1 --2- 172.21.15.177:0/2645547755 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe31809e7f0 0x7fe318122ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 1 -- 172.21.15.177:0/2645547755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318131710 msgr2=0x7fe318141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:29.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 1 --2- 172.21.15.177:0/2645547755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318131710 0x7fe318141750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:29.744 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 10 monclient: _finish_hunting 0 2024-09-15T03:58:29.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 1 monclient: found mon.noname-b 2024-09-15T03:58:29.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:29.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31d4f8640 1 -- 172.21.15.177:0/2645547755 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c009450 con 0x7fe3180a17d0 2024-09-15T03:58:29.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31e4fa640 10 monclient: get_auth_request con 0x7fe318131710 auth_method 0 2024-09-15T03:58:29.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.725+0000 7fe31e4fa640 1 --2- 172.21.15.177:0/2645547755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318131710 0x7fe318141750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:58:29.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.727+0000 7fe31d4f8640 1 -- 172.21.15.177:0/2645547755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3180a17d0 msgr2=0x7fe318141210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:29.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.727+0000 7fe31d4f8640 1 --2- 172.21.15.177:0/2645547755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3180a17d0 0x7fe318141210 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009210 tx=0x7fe30c032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:29.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe315ffb640 1 -- 172.21.15.177:0/2645547755 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c02aea0 con 0x7fe3180a17d0 2024-09-15T03:58:29.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe315ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe315ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:29.745 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe315ffb640 10 monclient: dump: 2024-09-15T03:58:29.745 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.745 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe315ffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe315ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe315ffb640 1 -- 172.21.15.177:0/2645547755 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30c000c40 con 0x7fe3180a17d0 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe31ff5d640 5 monclient: authenticate success, global_id 4126 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe31ff5d640 10 monclient: _renew_subs 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe31ff5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.746 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe31ff5d640 1 -- 172.21.15.177:0/2645547755 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31811fcf0 con 0x7fe3180a17d0 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe315ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe315ffb640 1 -- 172.21.15.177:0/2645547755 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c03d440 con 0x7fe3180a17d0 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe315ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe315ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe315ffb640 10 monclient: dump: 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:29.747 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe317fff640 4 set_mon_vals no callback set 2024-09-15T03:58:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe31ff5d640 10 monclient: _renew_subs 2024-09-15T03:58:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe31ff5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe31ff5d640 1 -- 172.21.15.177:0/2645547755 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3181218d0 con 0x7fe3180a17d0 2024-09-15T03:58:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe315ffb640 1 -- 172.21.15.177:0/2645547755 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fe30c039040 con 0x7fe3180a17d0 2024-09-15T03:58:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.728+0000 7fe315ffb640 1 -- 172.21.15.177:0/2645547755 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7fe30c03b3f0 con 0x7fe3180a17d0 2024-09-15T03:58:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.729+0000 7fe31ff5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.729+0000 7fe31ff5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.729+0000 7fe31ff5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.729+0000 7fe31ff5d640 1 -- 172.21.15.177:0/2645547755 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe318070950 con 0x7fe3180a17d0 2024-09-15T03:58:29.748 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:29.731+0000 7fe31d4f8640 1 --2- 172.21.15.177:0/2645547755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3180a17d0 0x7fe318141210 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009210 tx=0x7fe30c032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fe30c042070 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T03:58:30.537 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:30.535+0000 7fab3c215ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T03:58:30.560 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:30.559+0000 7fe315ffb640 1 -- 172.21.15.177:0/2645547755 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7fe30c042070 con 0x7fe3180a17d0 2024-09-15T03:58:30.561 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:30.559+0000 7fe315ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:30.561 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:30.559+0000 7fe315ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:30.587 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:30.586+0000 7fab3c215ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T03:58:30.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:30.626+0000 7fe31ff5d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0d65db82-4a0e-4f31-a10a-dcd71dcb269f", "id": 3}] 2024-09-15T03:58:30.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:30.626+0000 7fe31ff5d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0d65db82-4a0e-4f31-a10a-dcd71dcb269f", "id": 3}] 2024-09-15T03:58:30.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:30.626+0000 7fe31ff5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:30.628 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:30.626+0000 7fe31ff5d640 1 -- 172.21.15.177:0/2645547755 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0d65db82-4a0e-4f31-a10a-dcd71dcb269f", "id": 3} v 0) v1 -- 0x7fe3181208e0 con 0x7fe3180a17d0 2024-09-15T03:58:30.947 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:30.945+0000 7fab3c215ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T03:58:31.027 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:31.025+0000 7fab3c215ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T03:58:31.113 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:31.112+0000 7fab3c215ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T03:58:31.158 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:31.156+0000 7fab3c215ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T03:58:31.165 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.163+0000 7fe315ffb640 1 -- 172.21.15.177:0/2645547755 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0d65db82-4a0e-4f31-a10a-dcd71dcb269f", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe30c037070 con 0x7fe3180a17d0 2024-09-15T03:58:31.165 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.163+0000 7fe315ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0d65db82-4a0e-4f31-a10a-dcd71dcb269f", "id": 3}] 2024-09-15T03:58:31.165 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.163+0000 7fe315ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:31.165 INFO:teuthology.orchestra.run.smithi177.stdout:3 2024-09-15T03:58:31.166 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.164+0000 7fe31ff5d640 10 monclient: shutdown 2024-09-15T03:58:31.166 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.164+0000 7fe31ff5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:31.166 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.164+0000 7fe31ff5d640 1 -- 172.21.15.177:0/2645547755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3180a17d0 msgr2=0x7fe318141210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:31.166 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.164+0000 7fe31ff5d640 1 --2- 172.21.15.177:0/2645547755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3180a17d0 0x7fe318141210 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009210 tx=0x7fe30c032000 comp rx=0 tx=0).stop 2024-09-15T03:58:31.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.164+0000 7fe31ff5d640 1 -- 172.21.15.177:0/2645547755 shutdown_connections 2024-09-15T03:58:31.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.164+0000 7fe31ff5d640 1 --2- 172.21.15.177:0/2645547755 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe318131710 0x7fe318141750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.164+0000 7fe31ff5d640 1 --2- 172.21.15.177:0/2645547755 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3180a17d0 0x7fe318141210 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.165+0000 7fe31ff5d640 1 --2- 172.21.15.177:0/2645547755 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe31809e7f0 0x7fe318122ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.165+0000 7fe31ff5d640 1 -- 172.21.15.177:0/2645547755 >> 172.21.15.177:0/2645547755 conn(0x7fe3180631f0 msgr2=0x7fe3180a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:31.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.165+0000 7fe31ff5d640 1 -- 172.21.15.177:0/2645547755 shutdown_connections 2024-09-15T03:58:31.167 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.165+0000 7fe31ff5d640 1 -- 172.21.15.177:0/2645547755 wait complete. 2024-09-15T03:58:31.177 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new e95e4f77-578d-4d03-bb4d-ee72b43bc227 4 2024-09-15T03:58:31.259 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:31.257+0000 7fab3c215ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T03:58:31.270 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.268+0000 7f098bb5d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:31.270 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.268+0000 7f098bb5d640 10 monclient: build_initial_monmap 2024-09-15T03:58:31.270 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.268+0000 7f098bb5d640 10 monclient: monmap: 2024-09-15T03:58:31.270 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-09-15T03:58:31.270 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:31.270 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:31.270244+0000 2024-09-15T03:58:31.270 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:31.270244+0000 2024-09-15T03:58:31.270 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:31.270 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:31.270 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:31.270 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:31.271 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:31.271 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:31.271 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:31.271 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:31.273 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.269+0000 7f098bb5d640 1 Processor -- start 2024-09-15T03:58:31.273 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.269+0000 7f098bb5d640 1 -- start start 2024-09-15T03:58:31.273 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.269+0000 7f098bb5d640 10 monclient: init 2024-09-15T03:58:31.273 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.269+0000 7f098bb5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.269+0000 7f098bb5d640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T03:58:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.269+0000 7f098bb5d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f09840a17d0 0x7f098409f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.269+0000 7f098bb5d640 10 monclient(hunting): picked mon.noname-c con 0x7f09840a17d0 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.269+0000 7f098bb5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f098409e7f0 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.269+0000 7f098bb5d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0984136a40 0x7f0984138e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.269+0000 7f098bb5d640 10 monclient(hunting): picked mon.noname-b con 0x7f0984136a40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f098bb5d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09840a09e0 con 0x7f098409e7f0 2024-09-15T03:58:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f098bb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f098bb5d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09840a0760 con 0x7f09840a17d0 2024-09-15T03:58:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f098bb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f098bb5d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09840a1ba0 con 0x7f0984136a40 2024-09-15T03:58:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f098bb5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f098bb5d640 10 monclient(hunting): authenticate will time out at 735.322050s 2024-09-15T03:58:31.274 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f09840a17d0 0x7f098409f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f09840a17d0 0x7f098409f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:56174/0 (socket says 172.21.15.177:56174) 2024-09-15T03:58:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 1 -- 172.21.15.177:0/3290620552 learned_addr learned my addr 172.21.15.177:0/3290620552 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-09-15T03:58:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 10 monclient(hunting): get_auth_request con 0x7f09840a17d0 auth_method 0 2024-09-15T03:58:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f098a0fa640 1 --2- 172.21.15.177:0/3290620552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0984136a40 0x7f0984138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2024-09-15T03:58:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 1 -- 172.21.15.177:0/3290620552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0984136a40 msgr2=0x7f0984138e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 1 --2- 172.21.15.177:0/3290620552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0984136a40 0x7f0984138e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 1 -- 172.21.15.177:0/3290620552 >> v1:172.21.15.110:6789/0 conn(0x7f098409e7f0 legacy=0x7f098409ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:31.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 10 monclient: _finish_hunting 0 2024-09-15T03:58:31.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 1 monclient: found mon.noname-c 2024-09-15T03:58:31.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:31.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:58:31.276 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.270+0000 7f09898f9640 1 -- 172.21.15.177:0/3290620552 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098409ca20 con 0x7f09840a17d0 2024-09-15T03:58:31.282 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f09898f9640 1 -- 172.21.15.177:0/3290620552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09840a17d0 msgr2=0x7f098409f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:31.282 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f09898f9640 1 --2- 172.21.15.177:0/3290620552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09840a17d0 0x7f098409f840 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0980009940 tx=0x7f09800091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a44774560d32588f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:31.282 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f09888f7640 1 -- 172.21.15.177:0/3290620552 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0980032440 con 0x7f09840a17d0 2024-09-15T03:58:31.282 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f09888f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:31.282 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f09888f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:31.282 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f09888f7640 10 monclient: dump: 2024-09-15T03:58:31.282 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:31.282 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:31.283 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.283 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.283 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:31.283 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:31.283 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:31.283 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:31.283 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:31.283 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:31.283 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f09888f7640 10 monclient: _finish_auth 0 2024-09-15T03:58:31.283 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f09888f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:31.283 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f09888f7640 1 -- 172.21.15.177:0/3290620552 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09800325a0 con 0x7f09840a17d0 2024-09-15T03:58:31.283 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f098bb5d640 5 monclient: authenticate success, global_id 4129 2024-09-15T03:58:31.283 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f098bb5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:31.284 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f09888f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:31.284 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f098bb5d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:31.284 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f09888f7640 1 -- 172.21.15.177:0/3290620552 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0980032940 con 0x7f09840a17d0 2024-09-15T03:58:31.284 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f09888f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:31.284 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f098bb5d640 4 set_mon_vals no callback set 2024-09-15T03:58:31.284 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f09888f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:31.284 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f09888f7640 10 monclient: dump: 2024-09-15T03:58:31.284 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:31.284 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:31.284 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.284 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.284 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:31.284 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:31.284 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:31.285 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:31.285 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:31.285 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:31.285 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f098bb5d640 10 monclient: shutdown 2024-09-15T03:58:31.285 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f098bb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:31.285 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f098bb5d640 1 -- 172.21.15.177:0/3290620552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09840a17d0 msgr2=0x7f098409f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:31.285 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.273+0000 7f098bb5d640 1 --2- 172.21.15.177:0/3290620552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09840a17d0 0x7f098409f840 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0980009940 tx=0x7f09800091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.285 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.274+0000 7f098bb5d640 1 -- 172.21.15.177:0/3290620552 shutdown_connections 2024-09-15T03:58:31.285 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.274+0000 7f098bb5d640 1 --2- 172.21.15.177:0/3290620552 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0984136a40 0x7f0984138e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.285 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.274+0000 7f098bb5d640 1 --2- 172.21.15.177:0/3290620552 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f09840a17d0 0x7f098409f840 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.285 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.274+0000 7f098bb5d640 1 -- 172.21.15.177:0/3290620552 >> 172.21.15.177:0/3290620552 conn(0x7f09840631f0 msgr2=0x7f09840635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:31.285 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.274+0000 7f098bb5d640 1 -- 172.21.15.177:0/3290620552 shutdown_connections 2024-09-15T03:58:31.285 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.274+0000 7f098bb5d640 1 -- 172.21.15.177:0/3290620552 wait complete. 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.274+0000 7f098bb5d640 10 monclient: build_initial_monmap 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.274+0000 7f098bb5d640 10 monclient: monmap: 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 1 Processor -- start 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 1 -- start start 2024-09-15T03:58:31.286 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 10 monclient: init 2024-09-15T03:58:31.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:31.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:58:31.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f098409e7f0 0x7f0984133690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:31.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 10 monclient(hunting): picked mon.noname-c con 0x7f098409e7f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:31.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09840a17d0 0x7f098411c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:31.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 10 monclient(hunting): picked mon.noname-a con 0x7f09840a17d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:31.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0984136a40 0x7f098411c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:31.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 10 monclient(hunting): picked mon.noname-b con 0x7f0984136a40 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:31.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:31.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098411e1f0 con 0x7f09840a17d0 2024-09-15T03:58:31.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:31.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098411f7a0 con 0x7f0984136a40 2024-09-15T03:58:31.287 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:31.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098411dc70 con 0x7f098409e7f0 2024-09-15T03:58:31.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:31.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098bb5d640 10 monclient(hunting): authenticate will time out at 735.327561s 2024-09-15T03:58:31.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098a0fa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0984136a40 0x7f098411c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:31.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098a0fa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0984136a40 0x7f098411c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:56182/0 (socket says 172.21.15.177:56182) 2024-09-15T03:58:31.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09898f9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f098409e7f0 0x7f0984133690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:31.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098a0fa640 1 -- 172.21.15.177:0/1490509971 learned_addr learned my addr 172.21.15.177:0/1490509971 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-09-15T03:58:31.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098a0fa640 10 monclient(hunting): get_auth_request con 0x7f0984136a40 auth_method 0 2024-09-15T03:58:31.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098a0fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:31.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098a0fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:31.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098a0fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:31.288 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09898f9640 10 monclient(hunting): get_auth_request con 0x7f098409e7f0 auth_method 0 2024-09-15T03:58:31.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09898f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:31.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09898f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:31.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09898f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:31.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09890f8640 1 --2- 172.21.15.177:0/1490509971 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09840a17d0 0x7f098411c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:31.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09890f8640 10 monclient(hunting): get_auth_request con 0x7f09840a17d0 auth_method 0 2024-09-15T03:58:31.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09890f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:31.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09890f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:31.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09890f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:31.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098a0fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:31.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098a0fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:31.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098a0fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:31.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09898f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:31.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09898f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:31.289 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09898f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:31.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09890f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:31.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09890f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:31.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f09890f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:31.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.275+0000 7f098a0fa640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-09-15T03:58:31.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.276+0000 7f098a0fa640 1 -- 172.21.15.177:0/1490509971 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f098409e7f0 msgr2=0x7f0984133690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:31.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.276+0000 7f098a0fa640 1 --2- 172.21.15.177:0/1490509971 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f098409e7f0 0x7f0984133690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.276+0000 7f098a0fa640 1 -- 172.21.15.177:0/1490509971 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09840a17d0 msgr2=0x7f098411c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:31.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.276+0000 7f098a0fa640 1 --2- 172.21.15.177:0/1490509971 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09840a17d0 0x7f098411c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.276+0000 7f098a0fa640 10 monclient: _finish_hunting 0 2024-09-15T03:58:31.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.276+0000 7f098a0fa640 1 monclient: found mon.noname-b 2024-09-15T03:58:31.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.276+0000 7f098a0fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:31.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.276+0000 7f098a0fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-09-15T03:58:31.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.276+0000 7f098a0fa640 1 -- 172.21.15.177:0/1490509971 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0980009450 con 0x7f0984136a40 2024-09-15T03:58:31.290 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.276+0000 7f09890f8640 1 --2- 172.21.15.177:0/1490509971 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09840a17d0 0x7f098411c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:31.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.276+0000 7f09898f9640 1 --2- 172.21.15.177:0/1490509971 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f098409e7f0 0x7f0984133690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:31.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f098a0fa640 1 -- 172.21.15.177:0/1490509971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0984136a40 msgr2=0x7f098411c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:31.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f098a0fa640 1 --2- 172.21.15.177:0/1490509971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0984136a40 0x7f098411c680 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f097c00cee0 tx=0x7f097c007870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:31.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f0979ffb640 1 -- 172.21.15.177:0/1490509971 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097c00e610 con 0x7f0984136a40 2024-09-15T03:58:31.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f0979ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:31.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f0979ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:31.291 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f0979ffb640 10 monclient: dump: 2024-09-15T03:58:31.291 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:31.291 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:31.291 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.291 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.291 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:31.291 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:31.292 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:31.292 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:31.292 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:31.292 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:31.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f0979ffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:31.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f0979ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:31.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f0979ffb640 1 -- 172.21.15.177:0/1490509971 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097c00e770 con 0x7f0984136a40 2024-09-15T03:58:31.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f098bb5d640 5 monclient: authenticate success, global_id 4132 2024-09-15T03:58:31.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f0979ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:31.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f0979ffb640 1 -- 172.21.15.177:0/1490509971 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097c00eb30 con 0x7f0984136a40 2024-09-15T03:58:31.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f0979ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:31.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f0979ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:31.292 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f0979ffb640 10 monclient: dump: 2024-09-15T03:58:31.292 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:31.293 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:31.293 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.293 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.293 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:31.293 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:31.293 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:31.293 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:31.293 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:31.293 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:31.293 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f098bb5d640 10 monclient: _renew_subs 2024-09-15T03:58:31.293 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f098bb5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:31.293 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f097bfff640 4 set_mon_vals no callback set 2024-09-15T03:58:31.293 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f098bb5d640 1 -- 172.21.15.177:0/1490509971 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098409c250 con 0x7f0984136a40 2024-09-15T03:58:31.293 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f0979ffb640 1 -- 172.21.15.177:0/1490509971 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f097c009030 con 0x7f0984136a40 2024-09-15T03:58:31.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f098bb5d640 10 monclient: _renew_subs 2024-09-15T03:58:31.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f098bb5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:31.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f098bb5d640 1 -- 172.21.15.177:0/1490509971 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0984141d20 con 0x7f0984136a40 2024-09-15T03:58:31.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.278+0000 7f0979ffb640 1 -- 172.21.15.177:0/1490509971 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f097c012070 con 0x7f0984136a40 2024-09-15T03:58:31.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.279+0000 7f098bb5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:31.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.279+0000 7f098bb5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:31.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.279+0000 7f098bb5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:31.294 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.279+0000 7f098bb5d640 1 -- 172.21.15.177:0/1490509971 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0984137400 con 0x7f0984136a40 2024-09-15T03:58:31.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.281+0000 7f0979ffb640 1 -- 172.21.15.177:0/1490509971 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f0984141d20 con 0x7f0984136a40 2024-09-15T03:58:31.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.281+0000 7f0979ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:31.295 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.281+0000 7f0979ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:31.314 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:31.312+0000 7fab3c215ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T03:58:31.349 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.347+0000 7f098bb5d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e95e4f77-578d-4d03-bb4d-ee72b43bc227", "id": 4}] 2024-09-15T03:58:31.349 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.348+0000 7f098bb5d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e95e4f77-578d-4d03-bb4d-ee72b43bc227", "id": 4}] 2024-09-15T03:58:31.349 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.348+0000 7f098bb5d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:31.350 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.348+0000 7f098bb5d640 1 -- 172.21.15.177:0/1490509971 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e95e4f77-578d-4d03-bb4d-ee72b43bc227", "id": 4} v 0) v1 -- 0x7f0984070950 con 0x7f0984136a40 2024-09-15T03:58:31.354 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.352+0000 7f0979ffb640 1 -- 172.21.15.177:0/1490509971 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e95e4f77-578d-4d03-bb4d-ee72b43bc227", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f097c0168e0 con 0x7f0984136a40 2024-09-15T03:58:31.354 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.352+0000 7f0979ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e95e4f77-578d-4d03-bb4d-ee72b43bc227", "id": 4}] 2024-09-15T03:58:31.354 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.352+0000 7f0979ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:31.354 INFO:teuthology.orchestra.run.smithi177.stdout:4 2024-09-15T03:58:31.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.353+0000 7f098bb5d640 10 monclient: shutdown 2024-09-15T03:58:31.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.353+0000 7f098bb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:31.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.353+0000 7f098bb5d640 1 -- 172.21.15.177:0/1490509971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0984136a40 msgr2=0x7f098411c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:31.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.353+0000 7f098bb5d640 1 --2- 172.21.15.177:0/1490509971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0984136a40 0x7f098411c680 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f097c00cee0 tx=0x7f097c007870 comp rx=0 tx=0).stop 2024-09-15T03:58:31.355 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.353+0000 7f098bb5d640 1 -- 172.21.15.177:0/1490509971 shutdown_connections 2024-09-15T03:58:31.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.353+0000 7f098bb5d640 1 --2- 172.21.15.177:0/1490509971 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0984136a40 0x7f098411c680 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.353+0000 7f098bb5d640 1 --2- 172.21.15.177:0/1490509971 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f09840a17d0 0x7f098411c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.353+0000 7f098bb5d640 1 --2- 172.21.15.177:0/1490509971 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f098409e7f0 0x7f0984133690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.353+0000 7f098bb5d640 1 -- 172.21.15.177:0/1490509971 >> 172.21.15.177:0/1490509971 conn(0x7f09840631f0 msgr2=0x7f09840a4510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:31.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.353+0000 7f098bb5d640 1 -- 172.21.15.177:0/1490509971 shutdown_connections 2024-09-15T03:58:31.356 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.353+0000 7f098bb5d640 1 -- 172.21.15.177:0/1490509971 wait complete. 2024-09-15T03:58:31.359 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:31.357+0000 7fab3c215ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T03:58:31.365 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph --cluster ceph osd new f16ffee6-1e25-432c-b6c2-909a72ce24a5 5 2024-09-15T03:58:31.457 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:31.455+0000 7fab3c215ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T03:58:31.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.455+0000 7f0fa1d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:31.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.455+0000 7f0fa1d9d640 10 monclient: build_initial_monmap 2024-09-15T03:58:31.457 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.455+0000 7f0fa1d9d640 10 monclient: monmap: 2024-09-15T03:58:31.458 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-09-15T03:58:31.458 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:31.458 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:31.457505+0000 2024-09-15T03:58:31.458 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:31.457505+0000 2024-09-15T03:58:31.458 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:31.458 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:31.458 INFO:teuthology.orchestra.run.smithi177.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:31.458 INFO:teuthology.orchestra.run.smithi177.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:31.458 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:31.458 INFO:teuthology.orchestra.run.smithi177.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:31.458 INFO:teuthology.orchestra.run.smithi177.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:31.458 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:31.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.456+0000 7f0fa1d9d640 1 Processor -- start 2024-09-15T03:58:31.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.456+0000 7f0fa1d9d640 1 -- start start 2024-09-15T03:58:31.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.456+0000 7f0fa1d9d640 10 monclient: init 2024-09-15T03:58:31.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0fa1d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:31.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0fa1d9d640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-15T03:58:31.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0fa1d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0f9c0a17d0 0x7f0f9c09f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:31.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0fa1d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0f9c0a17d0 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:31.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0fa1d9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0f9c09e7f0 0x7f0f9c09ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:31.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0fa1d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0f9c09e7f0 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:31.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0fa1d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f9c131710 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:31.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0fa1d9d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c0a09e0 con 0x7f0f9c131710 2024-09-15T03:58:31.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0fa1d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:31.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0fa1d9d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9c0a0780 con 0x7f0f9c0a17d0 2024-09-15T03:58:31.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0fa1d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:31.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0fa1d9d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9c0a1ba0 con 0x7f0f9c09e7f0 2024-09-15T03:58:31.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0fa1d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:31.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0fa1d9d640 10 monclient(hunting): authenticate will time out at 735.509471s 2024-09-15T03:58:31.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0f9c09e7f0 0x7f0f9c09ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:31.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f0f9c09e7f0 0x7f0f9c09ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:56192/0 (socket says 172.21.15.177:56192) 2024-09-15T03:58:31.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 1 -- 172.21.15.177:0/1704018300 learned_addr learned my addr 172.21.15.177:0/1704018300 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-09-15T03:58:31.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 10 monclient(hunting): get_auth_request con 0x7f0f9c09e7f0 auth_method 0 2024-09-15T03:58:31.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:31.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:31.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:31.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f9b7fe640 1 --2- 172.21.15.177:0/1704018300 >> v2:172.21.15.110:3300/0 conn(0x7f0f9c0a17d0 0x7f0f9c09f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:31.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:31.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:31.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:31.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f9b7fe640 10 monclient(hunting): get_auth_request con 0x7f0f9c0a17d0 auth_method 0 2024-09-15T03:58:31.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:31.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:31.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:31.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:31.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:31.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:31.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 10 monclient(hunting): handle_auth_done global_id 4135 payload 274 2024-09-15T03:58:31.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 1 -- 172.21.15.177:0/1704018300 >> v2:172.21.15.110:3300/0 conn(0x7f0f9c0a17d0 msgr2=0x7f0f9c09f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:31.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 1 --2- 172.21.15.177:0/1704018300 >> v2:172.21.15.110:3300/0 conn(0x7f0f9c0a17d0 0x7f0f9c09f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 1 -- 172.21.15.177:0/1704018300 >> v1:172.21.15.110:6789/0 conn(0x7f0f9c131710 legacy=0x7f0f9c131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:31.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:31.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 1 monclient: found mon.noname-c 2024-09-15T03:58:31.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:31.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:58:31.464 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.457+0000 7f0f93fff640 1 -- 172.21.15.177:0/1704018300 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c09ca20 con 0x7f0f9c09e7f0 2024-09-15T03:58:31.470 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0f93fff640 1 -- 172.21.15.177:0/1704018300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f9c09e7f0 msgr2=0x7f0f9c09ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:31.470 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0f93fff640 1 --2- 172.21.15.177:0/1704018300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f9c09e7f0 0x7f0f9c09ec50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0f84009910 tx=0x7f0f840091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=606f37c449b115d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:31.471 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0f9affd640 1 -- 172.21.15.177:0/1704018300 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f84032440 con 0x7f0f9c09e7f0 2024-09-15T03:58:31.471 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0f9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:31.471 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0f9affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:31.471 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0f9affd640 10 monclient: dump: 2024-09-15T03:58:31.471 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:31.471 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:31.471 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.471 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.471 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:31.471 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:31.471 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:31.471 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:31.471 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:31.471 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:31.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0f9affd640 10 monclient: _finish_auth 0 2024-09-15T03:58:31.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0f9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:31.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0f9affd640 1 -- 172.21.15.177:0/1704018300 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f840325a0 con 0x7f0f9c09e7f0 2024-09-15T03:58:31.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0fa1d9d640 5 monclient: authenticate success, global_id 4135 2024-09-15T03:58:31.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0fa1d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:31.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0f9affd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:31.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0fa1d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:31.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0f9affd640 1 -- 172.21.15.177:0/1704018300 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f84032940 con 0x7f0f9c09e7f0 2024-09-15T03:58:31.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0fa1d9d640 4 set_mon_vals no callback set 2024-09-15T03:58:31.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0f9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:31.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0f9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:31.472 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0f9affd640 10 monclient: dump: 2024-09-15T03:58:31.472 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:31.472 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:31.473 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.473 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.473 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:31.473 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:31.473 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:31.473 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:31.474 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:31.474 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:31.474 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0fa1d9d640 10 monclient: shutdown 2024-09-15T03:58:31.474 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0fa1d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:31.474 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0fa1d9d640 1 -- 172.21.15.177:0/1704018300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f9c09e7f0 msgr2=0x7f0f9c09ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:31.474 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0fa1d9d640 1 --2- 172.21.15.177:0/1704018300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f9c09e7f0 0x7f0f9c09ec50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0f84009910 tx=0x7f0f840091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.474 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0fa1d9d640 1 -- 172.21.15.177:0/1704018300 shutdown_connections 2024-09-15T03:58:31.474 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0fa1d9d640 1 --2- 172.21.15.177:0/1704018300 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f9c09e7f0 0x7f0f9c09ec50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.474 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0fa1d9d640 1 --2- 172.21.15.177:0/1704018300 >> v2:172.21.15.110:3300/0 conn(0x7f0f9c0a17d0 0x7f0f9c09f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.474 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.461+0000 7f0fa1d9d640 1 -- 172.21.15.177:0/1704018300 >> 172.21.15.177:0/1704018300 conn(0x7f0f9c0631f0 msgr2=0x7f0f9c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:31.474 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.462+0000 7f0fa1d9d640 1 -- 172.21.15.177:0/1704018300 shutdown_connections 2024-09-15T03:58:31.474 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.462+0000 7f0fa1d9d640 1 -- 172.21.15.177:0/1704018300 wait complete. 2024-09-15T03:58:31.474 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.462+0000 7f0fa1d9d640 10 monclient: build_initial_monmap 2024-09-15T03:58:31.475 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.462+0000 7f0fa1d9d640 10 monclient: monmap: 2024-09-15T03:58:31.475 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 0 2024-09-15T03:58:31.475 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:31.475 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 0.000000 2024-09-15T03:58:31.475 INFO:teuthology.orchestra.run.smithi177.stderr:created 0.000000 2024-09-15T03:58:31.475 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:31.475 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:31.475 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:31.475 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:31.475 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:31.475 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:31.475 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.462+0000 7f0fa1d9d640 1 Processor -- start 2024-09-15T03:58:31.475 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.462+0000 7f0fa1d9d640 1 -- start start 2024-09-15T03:58:31.475 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.462+0000 7f0fa1d9d640 10 monclient: init 2024-09-15T03:58:31.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:31.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:58:31.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f9c09e7f0 0x7f0f9c1305f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:31.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f0f9c09e7f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:31.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f9c0a17d0 0x7f0f9c130b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:31.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f0f9c0a17d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:31.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f9c131710 0x7f0f9c11b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:31.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f0f9c131710 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:31.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:31.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c09c2c0 con 0x7f0f9c09e7f0 2024-09-15T03:58:31.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:31.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c11ea20 con 0x7f0f9c0a17d0 2024-09-15T03:58:31.476 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:31.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c1355d0 con 0x7f0f9c131710 2024-09-15T03:58:31.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:31.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0fa1d9d640 10 monclient(hunting): authenticate will time out at 735.515319s 2024-09-15T03:58:31.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f9c0a17d0 0x7f0f9c130b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:31.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f9c0a17d0 0x7f0f9c130b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:56208/0 (socket says 172.21.15.177:56208) 2024-09-15T03:58:31.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 1 -- 172.21.15.177:0/3955938612 learned_addr learned my addr 172.21.15.177:0/3955938612 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-09-15T03:58:31.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 10 monclient(hunting): get_auth_request con 0x7f0f9c0a17d0 auth_method 0 2024-09-15T03:58:31.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:31.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:31.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:31.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:31.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:31.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:31.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9b7fe640 1 --2- 172.21.15.177:0/3955938612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f9c09e7f0 0x7f0f9c1305f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:31.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9bfff640 1 --2- 172.21.15.177:0/3955938612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f9c131710 0x7f0f9c11b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:31.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9b7fe640 10 monclient(hunting): get_auth_request con 0x7f0f9c09e7f0 auth_method 0 2024-09-15T03:58:31.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:31.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:31.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:31.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9bfff640 10 monclient(hunting): get_auth_request con 0x7f0f9c131710 auth_method 0 2024-09-15T03:58:31.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:31.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:31.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:31.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:31.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:31.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:31.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:31.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:31.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:31.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-09-15T03:58:31.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 1 -- 172.21.15.177:0/3955938612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f9c131710 msgr2=0x7f0f9c11b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:31.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 1 --2- 172.21.15.177:0/3955938612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f9c131710 0x7f0f9c11b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 1 -- 172.21.15.177:0/3955938612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f9c09e7f0 msgr2=0x7f0f9c1305f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:31.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 1 --2- 172.21.15.177:0/3955938612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f9c09e7f0 0x7f0f9c1305f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:31.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:31.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 1 monclient: found mon.noname-b 2024-09-15T03:58:31.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:31.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-09-15T03:58:31.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.463+0000 7f0f93fff640 1 -- 172.21.15.177:0/3955938612 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f84009450 con 0x7f0f9c0a17d0 2024-09-15T03:58:31.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f93fff640 1 -- 172.21.15.177:0/3955938612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f9c0a17d0 msgr2=0x7f0f9c130b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:31.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f93fff640 1 --2- 172.21.15.177:0/3955938612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f9c0a17d0 0x7f0f9c130b30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0f84009970 tx=0x7f0f84032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:31.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f937fe640 1 -- 172.21.15.177:0/3955938612 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f84031a40 con 0x7f0f9c0a17d0 2024-09-15T03:58:31.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:31.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:31.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f937fe640 10 monclient: dump: 2024-09-15T03:58:31.480 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:31.480 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:31.480 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.480 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.480 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:31.480 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:31.481 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:31.481 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:31.481 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:31.481 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:31.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f937fe640 10 monclient: _finish_auth 0 2024-09-15T03:58:31.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:31.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f937fe640 1 -- 172.21.15.177:0/3955938612 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f84031bc0 con 0x7f0f9c0a17d0 2024-09-15T03:58:31.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0fa1d9d640 5 monclient: authenticate success, global_id 4138 2024-09-15T03:58:31.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f937fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:31.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0fa1d9d640 10 monclient: _renew_subs 2024-09-15T03:58:31.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f937fe640 1 -- 172.21.15.177:0/3955938612 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8403c400 con 0x7f0f9c0a17d0 2024-09-15T03:58:31.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0fa1d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:31.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0fa1d9d640 1 -- 172.21.15.177:0/3955938612 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c11e740 con 0x7f0f9c0a17d0 2024-09-15T03:58:31.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:31.482 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:31.482 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f9a7fc640 4 set_mon_vals no callback set 2024-09-15T03:58:31.482 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f937fe640 10 monclient: dump: 2024-09-15T03:58:31.482 INFO:teuthology.orchestra.run.smithi177.stderr:epoch 1 2024-09-15T03:58:31.482 INFO:teuthology.orchestra.run.smithi177.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:31.482 INFO:teuthology.orchestra.run.smithi177.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.482 INFO:teuthology.orchestra.run.smithi177.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:31.482 INFO:teuthology.orchestra.run.smithi177.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:31.482 INFO:teuthology.orchestra.run.smithi177.stderr:election_strategy: 1 2024-09-15T03:58:31.482 INFO:teuthology.orchestra.run.smithi177.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:31.482 INFO:teuthology.orchestra.run.smithi177.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:31.482 INFO:teuthology.orchestra.run.smithi177.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:31.482 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T03:58:31.482 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0fa1d9d640 10 monclient: _renew_subs 2024-09-15T03:58:31.483 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0fa1d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:31.483 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0fa1d9d640 1 -- 172.21.15.177:0/3955938612 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c135050 con 0x7f0f9c0a17d0 2024-09-15T03:58:31.483 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f937fe640 1 -- 172.21.15.177:0/3955938612 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f0f8403c830 con 0x7f0f9c0a17d0 2024-09-15T03:58:31.483 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.466+0000 7f0f937fe640 1 -- 172.21.15.177:0/3955938612 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f0f8403b920 con 0x7f0f9c0a17d0 2024-09-15T03:58:31.483 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.467+0000 7f0fa1d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:31.483 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.467+0000 7f0fa1d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:31.483 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.467+0000 7f0fa1d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:31.483 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.467+0000 7f0fa1d9d640 1 -- 172.21.15.177:0/3955938612 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c070950 con 0x7f0f9c0a17d0 2024-09-15T03:58:31.483 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.469+0000 7f0f937fe640 1 -- 172.21.15.177:0/3955938612 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f0f84037070 con 0x7f0f9c0a17d0 2024-09-15T03:58:31.483 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.469+0000 7f0f937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:31.483 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.469+0000 7f0f937fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:31.509 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:31.507+0000 7fab3c215ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T03:58:31.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.535+0000 7f0fa1d9d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f16ffee6-1e25-432c-b6c2-909a72ce24a5", "id": 5}] 2024-09-15T03:58:31.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.535+0000 7f0fa1d9d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f16ffee6-1e25-432c-b6c2-909a72ce24a5", "id": 5}] 2024-09-15T03:58:31.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.536+0000 7f0fa1d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:31.537 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:31.536+0000 7f0fa1d9d640 1 -- 172.21.15.177:0/3955938612 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f16ffee6-1e25-432c-b6c2-909a72ce24a5", "id": 5} v 0) v1 -- 0x7f0f9c11d800 con 0x7f0f9c0a17d0 2024-09-15T03:58:31.559 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:31.558+0000 7fab3c215ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T03:58:31.626 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:31.624+0000 7fab3c215ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T03:58:31.668 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:31.667+0000 7fab3c215ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T03:58:31.711 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:31.709+0000 7fab3c215ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T03:58:32.032 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:32.030+0000 7f0f937fe640 1 -- 172.21.15.177:0/3955938612 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f16ffee6-1e25-432c-b6c2-909a72ce24a5", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0f8403cd10 con 0x7f0f9c0a17d0 2024-09-15T03:58:32.032 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:32.030+0000 7f0f937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f16ffee6-1e25-432c-b6c2-909a72ce24a5", "id": 5}] 2024-09-15T03:58:32.032 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:32.030+0000 7f0f937fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:32.033 INFO:teuthology.orchestra.run.smithi177.stdout:5 2024-09-15T03:58:32.033 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:32.031+0000 7f0fa1d9d640 10 monclient: shutdown 2024-09-15T03:58:32.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:32.031+0000 7f0fa1d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:32.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:32.031+0000 7f0fa1d9d640 1 -- 172.21.15.177:0/3955938612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f9c0a17d0 msgr2=0x7f0f9c130b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:32.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:32.031+0000 7f0fa1d9d640 1 --2- 172.21.15.177:0/3955938612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f9c0a17d0 0x7f0f9c130b30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0f84009970 tx=0x7f0f84032000 comp rx=0 tx=0).stop 2024-09-15T03:58:32.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:32.031+0000 7f0fa1d9d640 1 -- 172.21.15.177:0/3955938612 shutdown_connections 2024-09-15T03:58:32.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:32.031+0000 7f0fa1d9d640 1 --2- 172.21.15.177:0/3955938612 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0f9c131710 0x7f0f9c11b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:32.031+0000 7f0fa1d9d640 1 --2- 172.21.15.177:0/3955938612 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0f9c0a17d0 0x7f0f9c130b30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:32.031+0000 7f0fa1d9d640 1 --2- 172.21.15.177:0/3955938612 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f9c09e7f0 0x7f0f9c1305f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:32.031+0000 7f0fa1d9d640 1 -- 172.21.15.177:0/3955938612 >> 172.21.15.177:0/3955938612 conn(0x7f0f9c0631f0 msgr2=0x7f0f9c1340c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:32.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:32.032+0000 7f0fa1d9d640 1 -- 172.21.15.177:0/3955938612 shutdown_connections 2024-09-15T03:58:32.034 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-15T03:58:32.032+0000 7f0fa1d9d640 1 -- 172.21.15.177:0/3955938612 wait complete. 2024-09-15T03:58:32.043 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-15T03:58:32.043 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-15T03:58:32.046 INFO:tasks.ceph.osd.0:Started 2024-09-15T03:58:32.046 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T03:58:32.046 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T03:58:32.048 INFO:tasks.ceph.osd.1:Started 2024-09-15T03:58:32.048 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-15T03:58:32.048 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-15T03:58:32.051 INFO:tasks.ceph.osd.2:Started 2024-09-15T03:58:32.051 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T03:58:32.051 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T03:58:32.053 INFO:tasks.ceph.osd.3:Started 2024-09-15T03:58:32.053 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-15T03:58:32.053 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-15T03:58:32.056 INFO:tasks.ceph.osd.4:Started 2024-09-15T03:58:32.056 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-15T03:58:32.056 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-15T03:58:32.058 INFO:tasks.ceph.osd.5:Started 2024-09-15T03:58:32.058 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:58:32.128 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-09-15T03:58:32.126+0000 7fda4e20a440 -1 Falling back to public interface 2024-09-15T03:58:32.128 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-15T03:58:32.126+0000 7f353b669440 -1 Falling back to public interface 2024-09-15T03:58:32.131 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-09-15T03:58:32.130+0000 7f02d1c8b440 -1 Falling back to public interface 2024-09-15T03:58:32.133 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-09-15T03:58:32.132+0000 7fc2e5924440 -1 Falling back to public interface 2024-09-15T03:58:32.135 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-09-15T03:58:32.134+0000 7f300e610440 -1 Falling back to public interface 2024-09-15T03:58:32.136 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-09-15T03:58:32.135+0000 7f5184f1d440 -1 Falling back to public interface 2024-09-15T03:58:32.137 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-09-15T03:58:32.136+0000 7fda4e20a440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:32.137 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-09-15T03:58:32.136+0000 7fda4e20a440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:58:32.137 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-09-15T03:58:32.136+0000 7fda4e20a440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:58:32.141 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-09-15T03:58:32.139+0000 7f02d1c8b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:32.141 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-09-15T03:58:32.139+0000 7f02d1c8b440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:58:32.141 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-09-15T03:58:32.139+0000 7f02d1c8b440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:58:32.142 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-15T03:58:32.140+0000 7f353b669440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:32.142 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-15T03:58:32.140+0000 7f353b669440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:58:32.142 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-15T03:58:32.140+0000 7f353b669440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:58:32.142 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-09-15T03:58:32.141+0000 7fda4e20a440 -1 osd.4 0 log_to_monitors true 2024-09-15T03:58:32.146 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-09-15T03:58:32.145+0000 7fc2e5924440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:32.146 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-09-15T03:58:32.145+0000 7fc2e5924440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:58:32.146 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-09-15T03:58:32.145+0000 7fc2e5924440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:58:32.146 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-09-15T03:58:32.145+0000 7f300e610440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:32.146 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-09-15T03:58:32.145+0000 7f300e610440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:58:32.147 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-09-15T03:58:32.145+0000 7f300e610440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:58:32.147 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-09-15T03:58:32.145+0000 7f02d1c8b440 -1 osd.5 0 log_to_monitors true 2024-09-15T03:58:32.147 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-09-15T03:58:32.146+0000 7f5184f1d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T03:58:32.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-09-15T03:58:32.147+0000 7f5184f1d440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:58:32.148 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-09-15T03:58:32.147+0000 7f5184f1d440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T03:58:32.152 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-09-15T03:58:32.151+0000 7f300e610440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:58:32.152 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-09-15T03:58:32.152+0000 7fc2e5924440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:58:32.153 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-09-15T03:58:32.152+0000 7f5184f1d440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:58:32.153 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-15T03:58:32.151+0000 7f353b669440 -1 osd.3 0 log_to_monitors true 2024-09-15T03:58:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.169+0000 7f625a6f8640 10 monclient: get_monmap_and_config 2024-09-15T03:58:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.169+0000 7f625a6f8640 10 monclient: build_initial_monmap 2024-09-15T03:58:32.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.169+0000 7f625a6f8640 10 monclient: monmap: 2024-09-15T03:58:32.171 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:32.171 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:32.171 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:32.170821+0000 2024-09-15T03:58:32.171 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:32.170821+0000 2024-09-15T03:58:32.171 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:32.171 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:32.171 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:32.171 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:32.171 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:32.171 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:32.171 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:32.172 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.170+0000 7f625a6f8640 1 Processor -- start 2024-09-15T03:58:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.170+0000 7f625a6f8640 1 -- start start 2024-09-15T03:58:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.170+0000 7f625a6f8640 10 monclient: init 2024-09-15T03:58:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f625a6f8640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f625a6f8640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-15T03:58:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f625a6f8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f62540a1a60 0x7f625409d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f625a6f8640 10 monclient(hunting): picked mon.noname-c con 0x7f62540a1a60 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:32.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f625a6f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f625409f870 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f625a6f8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62540a0380 0x7f6254138b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f625a6f8640 10 monclient(hunting): picked mon.noname-b con 0x7f62540a0380 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f625a6f8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62540a1260 con 0x7f625409f870 2024-09-15T03:58:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f625a6f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f625a6f8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f625409c870 con 0x7f62540a1a60 2024-09-15T03:58:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f625a6f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f625a6f8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62540a0c60 con 0x7f62540a0380 2024-09-15T03:58:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f625a6f8640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f625a6f8640 10 monclient(hunting): authenticate will time out at 735.815146s 2024-09-15T03:58:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62540a0380 0x7f6254138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6253fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f62540a1a60 0x7f625409d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:32.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f62537fe640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f625409f870 0x7f625409fc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.110:33772/0 (socket says 172.21.15.110:33772) 2024-09-15T03:58:32.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f62537fe640 1 -- 172.21.15.110:0/2580166262 learned_addr learned my addr 172.21.15.110:0/2580166262 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:32.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62540a0380 0x7f6254138b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:46332/0 (socket says 172.21.15.110:46332) 2024-09-15T03:58:32.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 10 monclient(hunting): get_auth_request con 0x7f62540a0380 auth_method 0 2024-09-15T03:58:32.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:32.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:32.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:32.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6253fff640 10 monclient(hunting): get_auth_request con 0x7f62540a1a60 auth_method 0 2024-09-15T03:58:32.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6253fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:32.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6253fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:32.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6253fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:32.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:32.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:32.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:32.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6253fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:32.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6253fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:32.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6253fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:32.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 10 monclient(hunting): handle_auth_done global_id 4157 payload 274 2024-09-15T03:58:32.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 1 -- 172.21.15.110:0/2580166262 >> v2:172.21.15.177:3300/0 conn(0x7f62540a1a60 msgr2=0x7f625409d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:32.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 1 --2- 172.21.15.110:0/2580166262 >> v2:172.21.15.177:3300/0 conn(0x7f62540a1a60 0x7f625409d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 1 -- 172.21.15.110:0/2580166262 >> v1:172.21.15.177:6789/0 conn(0x7f625409f870 legacy=0x7f625409fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:32.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 10 monclient: _finish_hunting 0 2024-09-15T03:58:32.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 1 monclient: found mon.noname-b 2024-09-15T03:58:32.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:32.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:32.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.171+0000 7f6258c95640 1 -- 172.21.15.110:0/2580166262 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625409cb10 con 0x7f62540a0380 2024-09-15T03:58:32.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f6258c95640 1 -- 172.21.15.110:0/2580166262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62540a0380 msgr2=0x7f6254138b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:32.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f6258c95640 1 --2- 172.21.15.110:0/2580166262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62540a0380 0x7f6254138b90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f624c009e60 tx=0x7f624c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24e23e13ef2ad6ea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:32.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f6252ffd640 1 -- 172.21.15.110:0/2580166262 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c007760 con 0x7f62540a0380 2024-09-15T03:58:32.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f6252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:32.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f6252ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:32.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f6252ffd640 10 monclient: dump: 2024-09-15T03:58:32.184 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:32.184 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:32.184 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.185 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.185 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:32.185 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:32.185 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:32.185 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:32.185 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:32.185 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:32.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f6252ffd640 10 monclient: _finish_auth 0 2024-09-15T03:58:32.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f6252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:32.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f625a6f8640 5 monclient: authenticate success, global_id 4157 2024-09-15T03:58:32.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f625a6f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:32.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f6252ffd640 1 -- 172.21.15.110:0/2580166262 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c02d050 con 0x7f62540a0380 2024-09-15T03:58:32.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f6252ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:32.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f625a6f8640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f625a6f8640 4 set_mon_vals no callback set 2024-09-15T03:58:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f6252ffd640 1 -- 172.21.15.110:0/2580166262 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c007af0 con 0x7f62540a0380 2024-09-15T03:58:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f6252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f6252ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.174+0000 7f6252ffd640 10 monclient: dump: 2024-09-15T03:58:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:32.186 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:32.187 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.175+0000 7f625a6f8640 10 monclient: shutdown 2024-09-15T03:58:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.175+0000 7f625a6f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.175+0000 7f625a6f8640 1 -- 172.21.15.110:0/2580166262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62540a0380 msgr2=0x7f6254138b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.175+0000 7f625a6f8640 1 --2- 172.21.15.110:0/2580166262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62540a0380 0x7f6254138b90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f624c009e60 tx=0x7f624c009780 comp rx=0 tx=0).stop 2024-09-15T03:58:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.175+0000 7f625a6f8640 1 -- 172.21.15.110:0/2580166262 shutdown_connections 2024-09-15T03:58:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.175+0000 7f625a6f8640 1 --2- 172.21.15.110:0/2580166262 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f62540a0380 0x7f6254138b90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.175+0000 7f625a6f8640 1 --2- 172.21.15.110:0/2580166262 >> v2:172.21.15.177:3300/0 conn(0x7f62540a1a60 0x7f625409d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.175+0000 7f625a6f8640 1 -- 172.21.15.110:0/2580166262 >> 172.21.15.110:0/2580166262 conn(0x7f6254063270 msgr2=0x7f6254063670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.175+0000 7f625a6f8640 1 -- 172.21.15.110:0/2580166262 shutdown_connections 2024-09-15T03:58:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.175+0000 7f625a6f8640 1 -- 172.21.15.110:0/2580166262 wait complete. 2024-09-15T03:58:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.175+0000 7f625a6f8640 10 monclient: build_initial_monmap 2024-09-15T03:58:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.175+0000 7f625a6f8640 10 monclient: monmap: 2024-09-15T03:58:32.187 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:32.188 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:32.188 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:32.188 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:32.188 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:32.188 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:32.188 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:32.188 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:32.188 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:32.188 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:32.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.175+0000 7f625a6f8640 1 Processor -- start 2024-09-15T03:58:32.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 1 -- start start 2024-09-15T03:58:32.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 10 monclient: init 2024-09-15T03:58:32.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:32.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:58:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f625409f870 0x7f6254140950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 10 monclient(hunting): picked mon.noname-c con 0x7f625409f870 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62540a0380 0x7f6254138700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 10 monclient(hunting): picked mon.noname-a con 0x7f62540a0380 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62540a1a60 0x7f625411e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 10 monclient(hunting): picked mon.noname-b con 0x7f62540a1a60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62540a1450 con 0x7f62540a0380 2024-09-15T03:58:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625409b7b0 con 0x7f62540a1a60 2024-09-15T03:58:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62541217e0 con 0x7f625409f870 2024-09-15T03:58:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:32.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f625a6f8640 10 monclient(hunting): authenticate will time out at 735.820620s 2024-09-15T03:58:32.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6253fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f625409f870 0x7f6254140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:32.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f62537fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62540a0380 0x7f6254138700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:32.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6253fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f625409f870 0x7f6254140950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:46342/0 (socket says 172.21.15.110:46342) 2024-09-15T03:58:32.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6253fff640 1 -- 172.21.15.110:0/3490905106 learned_addr learned my addr 172.21.15.110:0/3490905106 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:32.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6253fff640 10 monclient(hunting): get_auth_request con 0x7f625409f870 auth_method 0 2024-09-15T03:58:32.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6253fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:32.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6253fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:32.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6253fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:32.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f62537fe640 10 monclient(hunting): get_auth_request con 0x7f62540a0380 auth_method 0 2024-09-15T03:58:32.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6258c95640 1 --2- 172.21.15.110:0/3490905106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62540a1a60 0x7f625411e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:32.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f62537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:32.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f62537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:32.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f62537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:32.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6258c95640 10 monclient(hunting): get_auth_request con 0x7f62540a1a60 auth_method 0 2024-09-15T03:58:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6258c95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6258c95640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6258c95640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f62537fe640 1 -- 172.21.15.110:0/3490905106 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62540a0380 msgr2=0x7f6254138700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:58:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f62537fe640 1 -- 172.21.15.110:0/3490905106 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62540a0380 msgr2=0x7f6254138700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:58:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6253fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f62537fe640 1 --2- 172.21.15.110:0/3490905106 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62540a0380 0x7f6254138700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:58:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6253fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f62537fe640 1 --2- 172.21.15.110:0/3490905106 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62540a0380 0x7f6254138700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:58:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6253fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6258c95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6258c95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:32.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6258c95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.176+0000 7f6253fff640 10 monclient(hunting): handle_auth_done global_id 4160 payload 824 2024-09-15T03:58:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.177+0000 7f6253fff640 1 -- 172.21.15.110:0/3490905106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62540a1a60 msgr2=0x7f625411e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.177+0000 7f6253fff640 1 --2- 172.21.15.110:0/3490905106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62540a1a60 0x7f625411e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.177+0000 7f6253fff640 1 -- 172.21.15.110:0/3490905106 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62540a0380 msgr2=0x7f6254138700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:58:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.177+0000 7f6253fff640 1 --2- 172.21.15.110:0/3490905106 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62540a0380 0x7f6254138700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.177+0000 7f6253fff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.177+0000 7f6253fff640 1 monclient: found mon.noname-c 2024-09-15T03:58:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.177+0000 7f6253fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.177+0000 7f6253fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.177+0000 7f6253fff640 1 -- 172.21.15.110:0/3490905106 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624c009a20 con 0x7f625409f870 2024-09-15T03:58:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f6253fff640 1 -- 172.21.15.110:0/3490905106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f625409f870 msgr2=0x7f6254140950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f6253fff640 1 --2- 172.21.15.110:0/3490905106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f625409f870 0x7f6254140950 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f624400ea20 tx=0x7f62440090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:32.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f623bfff640 1 -- 172.21.15.110:0/3490905106 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6244009600 con 0x7f625409f870 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f623bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f623bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f623bfff640 10 monclient: dump: 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f623bfff640 10 monclient: _finish_auth 0 2024-09-15T03:58:32.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f623bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f623bfff640 1 -- 172.21.15.110:0/3490905106 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624400cc20 con 0x7f625409f870 2024-09-15T03:58:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f625a6f8640 5 monclient: authenticate success, global_id 4160 2024-09-15T03:58:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f623bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f623bfff640 1 -- 172.21.15.110:0/3490905106 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6244013470 con 0x7f625409f870 2024-09-15T03:58:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f623bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f623bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f623bfff640 10 monclient: dump: 2024-09-15T03:58:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:32.194 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f62527fc640 4 set_mon_vals no callback set 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f625a6f8640 10 monclient: _renew_subs 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f625a6f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f625a6f8640 1 -- 172.21.15.110:0/3490905106 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6254121b20 con 0x7f625409f870 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f625a6f8640 10 monclient: _renew_subs 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f625a6f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.179+0000 7f625a6f8640 1 -- 172.21.15.110:0/3490905106 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6254141840 con 0x7f625409f870 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.180+0000 7f623bfff640 1 -- 172.21.15.110:0/3490905106 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f6244005970 con 0x7f625409f870 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.180+0000 7f623bfff640 1 -- 172.21.15.110:0/3490905106 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f6244005c20 con 0x7f625409f870 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.180+0000 7f6239ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:32.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.180+0000 7f6239ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:32.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.180+0000 7f6239ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:32.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.180+0000 7f6239ffb640 1 -- 172.21.15.110:0/3490905106 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f622c005010 con 0x7f625409f870 2024-09-15T03:58:32.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.182+0000 7f623bfff640 1 -- 172.21.15.110:0/3490905106 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f6244021060 con 0x7f625409f870 2024-09-15T03:58:32.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.182+0000 7f623bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:32.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.182+0000 7f623bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.236+0000 7f6239ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.236+0000 7f6239ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.236+0000 7f6239ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.236+0000 7f6239ffb640 1 -- 172.21.15.110:0/3490905106 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f622c005590 con 0x7f625409f870 2024-09-15T03:58:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.237+0000 7f623bfff640 1 -- 172.21.15.110:0/3490905106 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f6244029020 con 0x7f625409f870 2024-09-15T03:58:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.237+0000 7f623bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:32.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.237+0000 7f623bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:32.239 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:32.239 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":8,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","created":"2024-09-15T03:58:28.265907+0000","modified":"2024-09-15T03:58:31.538597+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T03:58:31.538597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"105a3b3d-e410-47fb-ac54-6a5d5da4a46f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"77f0f205-6c64-495a-8edd-043de180efa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8a42031-303e-4ad0-9896-e5b84cc95cfb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0d65db82-4a0e-4f31-a10a-dcd71dcb269f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e95e4f77-578d-4d03-bb4d-ee72b43bc227","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f16ffee6-1e25-432c-b6c2-909a72ce24a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:58:32.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.238+0000 7f6239ffb640 10 monclient: shutdown 2024-09-15T03:58:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.238+0000 7f6239ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.238+0000 7f6239ffb640 1 -- 172.21.15.110:0/3490905106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f625409f870 msgr2=0x7f6254140950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.238+0000 7f6239ffb640 1 --2- 172.21.15.110:0/3490905106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f625409f870 0x7f6254140950 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f624400ea20 tx=0x7f62440090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.238+0000 7f6239ffb640 1 -- 172.21.15.110:0/3490905106 shutdown_connections 2024-09-15T03:58:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.238+0000 7f6239ffb640 1 --2- 172.21.15.110:0/3490905106 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f62540a1a60 0x7f625411e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.238+0000 7f6239ffb640 1 --2- 172.21.15.110:0/3490905106 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f62540a0380 0x7f6254138700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.238+0000 7f6239ffb640 1 --2- 172.21.15.110:0/3490905106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f625409f870 0x7f6254140950 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.238+0000 7f6239ffb640 1 -- 172.21.15.110:0/3490905106 >> 172.21.15.110:0/3490905106 conn(0x7f6254063270 msgr2=0x7f62540a3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.238+0000 7f6239ffb640 1 -- 172.21.15.110:0/3490905106 shutdown_connections 2024-09-15T03:58:32.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:32.238+0000 7f6239ffb640 1 -- 172.21.15.110:0/3490905106 wait complete. 2024-09-15T03:58:32.249 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-15T03:58:32.249 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-15T03:58:32.351 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:32.350+0000 7fab3c215ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T03:58:32.853 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T03:58:32.938 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.936+0000 7f1f2819d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:32.938 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.936+0000 7f1f2819d640 10 monclient: build_initial_monmap 2024-09-15T03:58:32.938 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.936+0000 7f1f2819d640 10 monclient: monmap: 2024-09-15T03:58:32.938 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-09-15T03:58:32.938 INFO:teuthology.misc.health.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:32.938 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-09-15T03:58:32.937758+0000 2024-09-15T03:58:32.938 INFO:teuthology.misc.health.smithi110.stderr:created 2024-09-15T03:58:32.937758+0000 2024-09-15T03:58:32.938 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:32.939 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:32.939 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:32.939 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:32.939 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:32.939 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:32.939 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:32.939 INFO:teuthology.misc.health.smithi110.stderr: 2024-09-15T03:58:32.939 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 1 Processor -- start 2024-09-15T03:58:32.939 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 1 -- start start 2024-09-15T03:58:32.939 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 10 monclient: init 2024-09-15T03:58:32.948 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:32.948 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-15T03:58:32.948 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1f2009b340 0x7f1f2009b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:32.949 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 10 monclient(hunting): picked mon.noname-a con 0x7f1f2009b340 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:32.949 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f2009bc50 0x7f1f2009c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:32.949 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 10 monclient(hunting): picked mon.noname-b con 0x7f1f2009bc50 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:32.949 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f2009c560 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:32.949 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2006a7d0 con 0x7f1f2009c560 2024-09-15T03:58:32.949 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:32.949 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2006c060 con 0x7f1f2009b340 2024-09-15T03:58:32.949 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:32.949 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.937+0000 7f1f2819d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2006bd80 con 0x7f1f2009bc50 2024-09-15T03:58:32.949 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f2819d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:32.949 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f2819d640 10 monclient(hunting): authenticate will time out at 736.582008s 2024-09-15T03:58:32.950 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1f2009b340 0x7f1f2009b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:32.950 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25738640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f2009bc50 0x7f1f2009c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:32.950 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f2673a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1f2009c560 0x7f1f2009c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:37920/0 (socket says 172.21.15.110:37920) 2024-09-15T03:58:32.950 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f2673a640 1 -- 172.21.15.110:0/1020155631 learned_addr learned my addr 172.21.15.110:0/1020155631 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:32.950 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 10 monclient(hunting): get_auth_request con 0x7f1f2009b340 auth_method 0 2024-09-15T03:58:32.950 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:32.950 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:32.950 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:32.950 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25738640 10 monclient(hunting): get_auth_request con 0x7f1f2009bc50 auth_method 0 2024-09-15T03:58:32.950 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:32.950 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25738640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:32.951 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25738640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:32.951 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:32.951 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:32.951 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:32.951 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:32.951 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:32.951 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:32.952 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 10 monclient(hunting): handle_auth_done global_id 4158 payload 274 2024-09-15T03:58:32.952 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 1 -- 172.21.15.110:0/1020155631 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f2009bc50 msgr2=0x7f1f2009c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:32.952 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 1 --2- 172.21.15.110:0/1020155631 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f2009bc50 0x7f1f2009c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.952 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 1 -- 172.21.15.110:0/1020155631 >> v1:172.21.15.110:6789/0 conn(0x7f1f2009c560 legacy=0x7f1f2009c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:32.952 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 10 monclient: _finish_hunting 0 2024-09-15T03:58:32.952 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 1 monclient: found mon.noname-a 2024-09-15T03:58:32.952 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:32.952 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:32.952 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25f39640 1 -- 172.21.15.110:0/1020155631 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2006ecd0 con 0x7f1f2009b340 2024-09-15T03:58:32.952 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.938+0000 7f1f25738640 1 --2- 172.21.15.110:0/1020155631 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f2009bc50 0x7f1f2009c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:32.952 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f25f39640 1 -- 172.21.15.110:0/1020155631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2009b340 msgr2=0x7f1f2009b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:32.953 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f25f39640 1 --2- 172.21.15.110:0/1020155631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2009b340 0x7f1f2009b710 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009850 tx=0x7f1f100090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e42c571b2567bafd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:32.953 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f24f37640 1 -- 172.21.15.110:0/1020155631 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1002ae80 con 0x7f1f2009b340 2024-09-15T03:58:32.953 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f24f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:32.953 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f24f37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:32.953 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f24f37640 10 monclient: dump: 2024-09-15T03:58:32.953 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-09-15T03:58:32.953 INFO:teuthology.misc.health.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:32.953 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.953 INFO:teuthology.misc.health.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.953 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:32.953 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:32.954 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:32.954 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:32.954 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:32.954 INFO:teuthology.misc.health.smithi110.stderr: 2024-09-15T03:58:32.954 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f24f37640 10 monclient: _finish_auth 0 2024-09-15T03:58:32.954 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f24f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:32.954 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f2819d640 5 monclient: authenticate success, global_id 4158 2024-09-15T03:58:32.954 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f2819d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:32.954 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f24f37640 1 -- 172.21.15.110:0/1020155631 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f10032440 con 0x7f1f2009b340 2024-09-15T03:58:32.955 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f24f37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:32.955 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f2819d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:32.955 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f2819d640 4 set_mon_vals no callback set 2024-09-15T03:58:32.955 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f24f37640 1 -- 172.21.15.110:0/1020155631 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f100327c0 con 0x7f1f2009b340 2024-09-15T03:58:32.955 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f24f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:32.955 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f24f37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:32.955 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f24f37640 10 monclient: dump: 2024-09-15T03:58:32.955 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-09-15T03:58:32.955 INFO:teuthology.misc.health.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:32.955 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.956 INFO:teuthology.misc.health.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.956 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:32.956 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:32.956 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:32.956 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:32.956 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:32.956 INFO:teuthology.misc.health.smithi110.stderr: 2024-09-15T03:58:32.956 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f2819d640 10 monclient: shutdown 2024-09-15T03:58:32.956 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f2819d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:32.956 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f2819d640 1 -- 172.21.15.110:0/1020155631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2009b340 msgr2=0x7f1f2009b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:32.957 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f2819d640 1 --2- 172.21.15.110:0/1020155631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2009b340 0x7f1f2009b710 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009850 tx=0x7f1f100090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.957 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f2819d640 1 -- 172.21.15.110:0/1020155631 shutdown_connections 2024-09-15T03:58:32.957 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f2819d640 1 --2- 172.21.15.110:0/1020155631 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f2009bc50 0x7f1f2009c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.957 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f2819d640 1 --2- 172.21.15.110:0/1020155631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2009b340 0x7f1f2009b710 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.957 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.941+0000 7f1f2819d640 1 -- 172.21.15.110:0/1020155631 >> 172.21.15.110:0/1020155631 conn(0x7f1f2009d7b0 msgr2=0x7f1f2009dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:32.957 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.942+0000 7f1f2819d640 1 -- 172.21.15.110:0/1020155631 shutdown_connections 2024-09-15T03:58:32.957 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.942+0000 7f1f2819d640 1 -- 172.21.15.110:0/1020155631 wait complete. 2024-09-15T03:58:32.957 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.942+0000 7f1f2819d640 10 monclient: build_initial_monmap 2024-09-15T03:58:32.957 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.942+0000 7f1f2819d640 10 monclient: monmap: 2024-09-15T03:58:32.957 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-09-15T03:58:32.957 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:32.958 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:32.958 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-09-15T03:58:32.958 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:32.958 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:32.958 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:32.958 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:32.958 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:32.958 INFO:teuthology.misc.health.smithi110.stderr: 2024-09-15T03:58:32.958 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.942+0000 7f1f2819d640 1 Processor -- start 2024-09-15T03:58:32.958 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.942+0000 7f1f2819d640 1 -- start start 2024-09-15T03:58:32.958 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.942+0000 7f1f2819d640 10 monclient: init 2024-09-15T03:58:32.958 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:32.959 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:58:32.959 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2009b340 0x7f1f2012da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:32.959 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 10 monclient(hunting): picked mon.noname-a con 0x7f1f2009b340 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:32.959 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f2009bc50 0x7f1f20131600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:32.959 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 10 monclient(hunting): picked mon.noname-b con 0x7f1f2009bc50 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:32.959 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f2009c560 0x7f1f20131b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:32.959 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 10 monclient(hunting): picked mon.noname-c con 0x7f1f2009c560 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:32.959 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:32.959 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2006b260 con 0x7f1f2009b340 2024-09-15T03:58:32.959 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:32.959 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f20099df0 con 0x7f1f2009bc50 2024-09-15T03:58:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f20134c10 con 0x7f1f2009c560 2024-09-15T03:58:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2819d640 10 monclient(hunting): authenticate will time out at 736.587284s 2024-09-15T03:58:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2009b340 0x7f1f2012da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2673a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f2009c560 0x7f1f20131b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2009b340 0x7f1f2012da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56096/0 (socket says 172.21.15.110:56096) 2024-09-15T03:58:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 1 -- 172.21.15.110:0/3639510635 learned_addr learned my addr 172.21.15.110:0/3639510635 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 10 monclient(hunting): get_auth_request con 0x7f1f2009b340 auth_method 0 2024-09-15T03:58:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:32.960 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2673a640 10 monclient(hunting): get_auth_request con 0x7f1f2009c560 auth_method 0 2024-09-15T03:58:32.961 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2673a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:32.961 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2673a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:32.961 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2673a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:32.961 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:32.961 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:32.961 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:32.961 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2673a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:32.961 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2673a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:32.961 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2673a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:32.961 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25738640 1 --2- 172.21.15.110:0/3639510635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f2009bc50 0x7f1f20131600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:32.962 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25738640 10 monclient(hunting): get_auth_request con 0x7f1f2009bc50 auth_method 0 2024-09-15T03:58:32.962 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:32.962 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25738640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:32.962 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25738640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:32.962 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:32.962 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:32.962 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:32.962 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 10 monclient(hunting): handle_auth_done global_id 4161 payload 824 2024-09-15T03:58:32.962 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 1 -- 172.21.15.110:0/3639510635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f2009c560 msgr2=0x7f1f20131b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:32.962 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 1 --2- 172.21.15.110:0/3639510635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f2009c560 0x7f1f20131b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.962 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 1 -- 172.21.15.110:0/3639510635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f2009bc50 msgr2=0x7f1f20131600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:32.963 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 1 --2- 172.21.15.110:0/3639510635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f2009bc50 0x7f1f20131600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:32.963 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 10 monclient: _finish_hunting 0 2024-09-15T03:58:32.963 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 1 monclient: found mon.noname-a 2024-09-15T03:58:32.963 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:32.963 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:32.963 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f25f39640 1 -- 172.21.15.110:0/3639510635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c009390 con 0x7f1f2009b340 2024-09-15T03:58:32.963 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.943+0000 7f1f2673a640 1 --2- 172.21.15.110:0/3639510635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f2009c560 0x7f1f20131b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:32.963 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.945+0000 7f1f25f39640 1 -- 172.21.15.110:0/3639510635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2009b340 msgr2=0x7f1f2012da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:32.963 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.945+0000 7f1f25f39640 1 --2- 172.21.15.110:0/3639510635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2009b340 0x7f1f2012da70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009850 tx=0x7f1f10032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:32.963 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f15ffb640 1 -- 172.21.15.110:0/3639510635 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10031c70 con 0x7f1f2009b340 2024-09-15T03:58:32.964 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:32.964 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f15ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:32.964 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f15ffb640 10 monclient: dump: 2024-09-15T03:58:32.964 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-09-15T03:58:32.964 INFO:teuthology.misc.health.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:32.964 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.964 INFO:teuthology.misc.health.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.964 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:32.964 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:32.964 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:32.964 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:32.964 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:32.965 INFO:teuthology.misc.health.smithi110.stderr: 2024-09-15T03:58:32.965 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f15ffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:32.965 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:32.965 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f2819d640 5 monclient: authenticate success, global_id 4161 2024-09-15T03:58:32.965 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f15ffb640 1 -- 172.21.15.110:0/3639510635 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f10031dd0 con 0x7f1f2009b340 2024-09-15T03:58:32.965 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f15ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:32.965 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f15ffb640 1 -- 172.21.15.110:0/3639510635 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1003d650 con 0x7f1f2009b340 2024-09-15T03:58:32.965 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f2819d640 10 monclient: _renew_subs 2024-09-15T03:58:32.965 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f2819d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:32.965 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f2819d640 1 -- 172.21.15.110:0/3639510635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f10009390 con 0x7f1f2009b340 2024-09-15T03:58:32.965 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:32.966 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:32.966 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f15ffb640 10 monclient: dump: 2024-09-15T03:58:32.966 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-09-15T03:58:32.966 INFO:teuthology.misc.health.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:32.966 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.966 INFO:teuthology.misc.health.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:32.966 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:32.966 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:32.966 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:32.966 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:32.966 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:32.967 INFO:teuthology.misc.health.smithi110.stderr: 2024-09-15T03:58:32.967 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f17fff640 4 set_mon_vals no callback set 2024-09-15T03:58:32.967 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f2819d640 10 monclient: _renew_subs 2024-09-15T03:58:32.967 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f2819d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:32.967 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f2819d640 1 -- 172.21.15.110:0/3639510635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f201330c0 con 0x7f1f2009b340 2024-09-15T03:58:32.967 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f25f39640 1 --2- 172.21.15.110:0/3639510635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2009b340 0x7f1f2012da70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009850 tx=0x7f1f10032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f1f10039040 mgrmap(e 1) v1 2024-09-15T03:58:32.967 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.946+0000 7f1f15ffb640 1 -- 172.21.15.110:0/3639510635 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f1f10039040 con 0x7f1f2009b340 2024-09-15T03:58:32.967 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.947+0000 7f1f2819d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:32.967 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.947+0000 7f1f2819d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:32.967 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.947+0000 7f1f2819d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:32.967 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.947+0000 7f1f2819d640 1 -- 172.21.15.110:0/3639510635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f200647e0 con 0x7f1f2009b340 2024-09-15T03:58:32.968 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:32.949+0000 7f1f15ffb640 1 -- 172.21.15.110:0/3639510635 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f1f10047a40 con 0x7f1f2009b340 2024-09-15T03:58:33.775 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.774+0000 7f1f15ffb640 1 -- 172.21.15.110:0/3639510635 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f1f1004c060 con 0x7f1f2009b340 2024-09-15T03:58:33.775 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.774+0000 7f1f15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:33.775 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.774+0000 7f1f15ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:33.830 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.829+0000 7f1f2819d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:33.831 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.829+0000 7f1f2819d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:33.831 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.829+0000 7f1f2819d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:33.831 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.829+0000 7f1f2819d640 1 -- 172.21.15.110:0/3639510635 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f2012dfb0 con 0x7f1f2009b340 2024-09-15T03:58:33.831 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.830+0000 7f1f15ffb640 1 -- 172.21.15.110:0/3639510635 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v9) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f1f10042070 con 0x7f1f2009b340 2024-09-15T03:58:33.831 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.830+0000 7f1f15ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:33.831 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.830+0000 7f1f15ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:33.831 INFO:teuthology.misc.health.smithi110.stdout: 2024-09-15T03:58:33.831 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":9,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","created":"2024-09-15T03:58:28.265907+0000","modified":"2024-09-15T03:58:33.030440+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T03:58:31.538597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"105a3b3d-e410-47fb-ac54-6a5d5da4a46f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"77f0f205-6c64-495a-8edd-043de180efa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e8a42031-303e-4ad0-9896-e5b84cc95cfb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0d65db82-4a0e-4f31-a10a-dcd71dcb269f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e95e4f77-578d-4d03-bb4d-ee72b43bc227","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f16ffee6-1e25-432c-b6c2-909a72ce24a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:58:33.832 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.830+0000 7f1f2819d640 10 monclient: shutdown 2024-09-15T03:58:33.832 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.830+0000 7f1f2819d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:33.833 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.830+0000 7f1f2819d640 1 -- 172.21.15.110:0/3639510635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2009b340 msgr2=0x7f1f2012da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:33.833 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.830+0000 7f1f2819d640 1 --2- 172.21.15.110:0/3639510635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2009b340 0x7f1f2012da70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009850 tx=0x7f1f10032000 comp rx=0 tx=0).stop 2024-09-15T03:58:33.833 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.831+0000 7f1f2819d640 1 -- 172.21.15.110:0/3639510635 shutdown_connections 2024-09-15T03:58:33.833 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.831+0000 7f1f2819d640 1 --2- 172.21.15.110:0/3639510635 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1f2009c560 0x7f1f20131b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:33.833 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.831+0000 7f1f2819d640 1 --2- 172.21.15.110:0/3639510635 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1f2009bc50 0x7f1f20131600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:33.834 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.831+0000 7f1f2819d640 1 --2- 172.21.15.110:0/3639510635 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f2009b340 0x7f1f2012da70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:33.834 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.831+0000 7f1f2819d640 1 -- 172.21.15.110:0/3639510635 >> 172.21.15.110:0/3639510635 conn(0x7f1f2009d7b0 msgr2=0x7f1f2009f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:33.834 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.831+0000 7f1f2819d640 1 -- 172.21.15.110:0/3639510635 shutdown_connections 2024-09-15T03:58:33.834 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:33.831+0000 7f1f2819d640 1 -- 172.21.15.110:0/3639510635 wait complete. 2024-09-15T03:58:33.841 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-15T03:58:34.035 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-09-15T03:58:34.034+0000 7fc2e0bf7640 -1 osd.1 0 waiting for initial osdmap 2024-09-15T03:58:34.035 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-09-15T03:58:34.034+0000 7f51801f7640 -1 osd.0 0 waiting for initial osdmap 2024-09-15T03:58:34.035 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-09-15T03:58:34.034+0000 7f30095f7640 -1 osd.2 0 waiting for initial osdmap 2024-09-15T03:58:34.036 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-09-15T03:58:34.033+0000 7fda491f7640 -1 osd.4 0 waiting for initial osdmap 2024-09-15T03:58:34.036 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-15T03:58:34.034+0000 7f3536ff7640 -1 osd.3 0 waiting for initial osdmap 2024-09-15T03:58:34.036 INFO:tasks.ceph.osd.1.smithi110.stderr:2024-09-15T03:58:34.036+0000 7fc2dd28c640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:58:34.037 INFO:tasks.ceph.osd.0.smithi110.stderr:2024-09-15T03:58:34.036+0000 7f517c9f0640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:58:34.037 INFO:tasks.ceph.osd.2.smithi110.stderr:2024-09-15T03:58:34.036+0000 7f3005df0640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:58:34.037 INFO:tasks.ceph.osd.3.smithi177.stderr:2024-09-15T03:58:34.035+0000 7f35337f0640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:58:34.039 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-09-15T03:58:34.038+0000 7fda491f7640 -1 osd.4 0 waiting for initial osdmap 2024-09-15T03:58:34.039 INFO:tasks.ceph.osd.4.smithi177.stderr:2024-09-15T03:58:34.038+0000 7fda4588c640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:58:34.233 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-09-15T03:58:34.232+0000 7f02cd5f7640 -1 osd.5 0 waiting for initial osdmap 2024-09-15T03:58:34.235 INFO:tasks.ceph.osd.5.smithi177.stderr:2024-09-15T03:58:34.234+0000 7f02c9c8c640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:58:35.689 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:35.688+0000 7fab20df6640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-15T03:58:35.696 INFO:tasks.ceph.mgr.x.smithi177.stderr:2024-09-15T03:58:35.694+0000 7fab20df6640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-15T03:58:40.448 DEBUG:teuthology.orchestra.run.smithi110:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T03:58:40.543 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.542+0000 7f816191d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:40.543 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.542+0000 7f816191d640 10 monclient: build_initial_monmap 2024-09-15T03:58:40.543 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.542+0000 7f816191d640 10 monclient: monmap: 2024-09-15T03:58:40.543 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-09-15T03:58:40.543 INFO:teuthology.misc.health.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:40.543 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-09-15T03:58:40.543222+0000 2024-09-15T03:58:40.543 INFO:teuthology.misc.health.smithi110.stderr:created 2024-09-15T03:58:40.543222+0000 2024-09-15T03:58:40.543 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:40.543 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:40.543 INFO:teuthology.misc.health.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:40.543 INFO:teuthology.misc.health.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:40.543 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:40.544 INFO:teuthology.misc.health.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:40.544 INFO:teuthology.misc.health.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:40.544 INFO:teuthology.misc.health.smithi110.stderr: 2024-09-15T03:58:40.547 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.542+0000 7f816191d640 1 Processor -- start 2024-09-15T03:58:40.547 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.542+0000 7f816191d640 1 -- start start 2024-09-15T03:58:40.547 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.542+0000 7f816191d640 10 monclient: init 2024-09-15T03:58:40.547 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f816191d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:40.547 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f816191d640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T03:58:40.547 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f816191d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f815c09e650 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:40.547 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f816191d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c0a19d0 0x7f815c09ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:40.547 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f816191d640 10 monclient(hunting): picked mon.noname-b con 0x7f815c0a19d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:40.547 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f816191d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f815c09ef60 0x7f815c1389d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:40.548 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f816191d640 10 monclient(hunting): picked mon.noname-c con 0x7f815c09ef60 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:40.548 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f816191d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815c09c880 con 0x7f815c09e650 2024-09-15T03:58:40.548 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f816191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:40.548 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f816191d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f815c0a0810 con 0x7f815c09ef60 2024-09-15T03:58:40.548 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f816191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:40.548 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f816191d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f815c09cb60 con 0x7f815c0a19d0 2024-09-15T03:58:40.548 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f816191d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:40.548 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f816191d640 10 monclient(hunting): authenticate will time out at 744.187426s 2024-09-15T03:58:40.548 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815a7fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c0a19d0 0x7f815c09ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:40.548 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815a7fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c0a19d0 0x7f815c09ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54776/0 (socket says 172.21.15.110:54776) 2024-09-15T03:58:40.548 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815a7fc640 1 -- 172.21.15.110:0/3255676384 learned_addr learned my addr 172.21.15.110:0/3255676384 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:40.548 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815a7fc640 10 monclient(hunting): get_auth_request con 0x7f815c0a19d0 auth_method 0 2024-09-15T03:58:40.548 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:40.549 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:40.549 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:40.549 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:40.549 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:40.549 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:40.549 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815b7fe640 1 --2- 172.21.15.110:0/3255676384 >> v2:172.21.15.177:3300/0 conn(0x7f815c09ef60 0x7f815c1389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:40.549 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815b7fe640 10 monclient(hunting): get_auth_request con 0x7f815c09ef60 auth_method 0 2024-09-15T03:58:40.549 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:40.549 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:40.549 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:40.549 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815a7fc640 1 -- 172.21.15.110:0/3255676384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c0a19d0 msgr2=0x7f815c09ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T03:58:40.549 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815a7fc640 1 -- 172.21.15.110:0/3255676384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c0a19d0 msgr2=0x7f815c09ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:58:40.549 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815a7fc640 1 --2- 172.21.15.110:0/3255676384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c0a19d0 0x7f815c09ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:58:40.549 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815a7fc640 1 --2- 172.21.15.110:0/3255676384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c0a19d0 0x7f815c09ea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:58:40.550 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:40.550 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:40.550 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.543+0000 7f815b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:40.550 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.544+0000 7f815b7fe640 10 monclient(hunting): handle_auth_done global_id 4195 payload 274 2024-09-15T03:58:40.550 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.544+0000 7f815b7fe640 1 -- 172.21.15.110:0/3255676384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c0a19d0 msgr2=0x7f815c09ea20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:58:40.550 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.544+0000 7f815b7fe640 1 --2- 172.21.15.110:0/3255676384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c0a19d0 0x7f815c09ea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.550 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.544+0000 7f815b7fe640 1 -- 172.21.15.110:0/3255676384 >> v1:172.21.15.177:6789/0 conn(0x7f815c09e650 legacy=0x7f815c131a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:40.550 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.544+0000 7f815b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:58:40.550 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.544+0000 7f815b7fe640 1 monclient: found mon.noname-c 2024-09-15T03:58:40.550 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.544+0000 7f815b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:40.550 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.544+0000 7f815b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:58:40.550 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.544+0000 7f815b7fe640 1 -- 172.21.15.110:0/3255676384 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815c09c340 con 0x7f815c09ef60 2024-09-15T03:58:40.557 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.546+0000 7f815b7fe640 1 -- 172.21.15.110:0/3255676384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f815c09ef60 msgr2=0x7f815c1389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:40.557 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f815b7fe640 1 --2- 172.21.15.110:0/3255676384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f815c09ef60 0x7f815c1389d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f814c009850 tx=0x7f814c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d7b925fa1a18ac6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:40.557 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f8159ffb640 1 -- 172.21.15.110:0/3255676384 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814c02adf0 con 0x7f815c09ef60 2024-09-15T03:58:40.557 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f8159ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:40.557 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f8159ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:40.557 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f8159ffb640 10 monclient: dump: 2024-09-15T03:58:40.557 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-09-15T03:58:40.557 INFO:teuthology.misc.health.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:40.557 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.557 INFO:teuthology.misc.health.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.557 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:40.558 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:40.558 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:40.558 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:40.558 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:40.558 INFO:teuthology.misc.health.smithi110.stderr: 2024-09-15T03:58:40.558 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f8159ffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:40.558 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f8159ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f816191d640 5 monclient: authenticate success, global_id 4195 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f816191d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f8159ffb640 1 -- 172.21.15.110:0/3255676384 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f814c032440 con 0x7f815c09ef60 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f8159ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f816191d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f816191d640 4 set_mon_vals no callback set 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f8159ffb640 1 -- 172.21.15.110:0/3255676384 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814c032820 con 0x7f815c09ef60 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f8159ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f8159ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f8159ffb640 10 monclient: dump: 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.559 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:40.560 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:40.560 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:40.560 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:40.560 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:40.560 INFO:teuthology.misc.health.smithi110.stderr: 2024-09-15T03:58:40.560 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f816191d640 10 monclient: shutdown 2024-09-15T03:58:40.560 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f816191d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:40.560 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f816191d640 1 -- 172.21.15.110:0/3255676384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f815c09ef60 msgr2=0x7f815c1389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:40.560 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f816191d640 1 --2- 172.21.15.110:0/3255676384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f815c09ef60 0x7f815c1389d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f814c009850 tx=0x7f814c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.560 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.547+0000 7f816191d640 1 -- 172.21.15.110:0/3255676384 shutdown_connections 2024-09-15T03:58:40.560 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.548+0000 7f816191d640 1 --2- 172.21.15.110:0/3255676384 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f815c09ef60 0x7f815c1389d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.560 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.548+0000 7f816191d640 1 --2- 172.21.15.110:0/3255676384 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c0a19d0 0x7f815c09ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.560 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.548+0000 7f816191d640 1 -- 172.21.15.110:0/3255676384 >> 172.21.15.110:0/3255676384 conn(0x7f815c069750 msgr2=0x7f815c06e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:40.561 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.548+0000 7f816191d640 1 -- 172.21.15.110:0/3255676384 shutdown_connections 2024-09-15T03:58:40.561 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.548+0000 7f816191d640 1 -- 172.21.15.110:0/3255676384 wait complete. 2024-09-15T03:58:40.561 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.548+0000 7f816191d640 10 monclient: build_initial_monmap 2024-09-15T03:58:40.561 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.548+0000 7f816191d640 10 monclient: monmap: 2024-09-15T03:58:40.561 INFO:teuthology.misc.health.smithi110.stderr:epoch 0 2024-09-15T03:58:40.561 INFO:teuthology.misc.health.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:40.561 INFO:teuthology.misc.health.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:40.561 INFO:teuthology.misc.health.smithi110.stderr:created 0.000000 2024-09-15T03:58:40.562 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:40.562 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:40.562 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:40.562 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:40.562 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:40.562 INFO:teuthology.misc.health.smithi110.stderr: 2024-09-15T03:58:40.562 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 1 Processor -- start 2024-09-15T03:58:40.562 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 1 -- start start 2024-09-15T03:58:40.563 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 10 monclient: init 2024-09-15T03:58:40.563 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:40.563 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:58:40.563 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c09e650 0x7f815c138140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:40.563 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 10 monclient(hunting): picked mon.noname-c con 0x7f815c09e650 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:40.563 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f815c09ef60 0x7f815c133eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:40.563 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 10 monclient(hunting): picked mon.noname-b con 0x7f815c09ef60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:40.563 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f815c0a19d0 0x7f815c1343f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:40.563 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 10 monclient(hunting): picked mon.noname-a con 0x7f815c0a19d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:40.563 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:40.563 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c137480 con 0x7f815c0a19d0 2024-09-15T03:58:40.563 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:40.563 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c138680 con 0x7f815c09ef60 2024-09-15T03:58:40.563 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:40.564 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f815c135ed0 con 0x7f815c09e650 2024-09-15T03:58:40.564 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:40.564 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f816191d640 10 monclient(hunting): authenticate will time out at 744.193517s 2024-09-15T03:58:40.564 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c09e650 0x7f815c138140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:40.564 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815b7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f815c0a19d0 0x7f815c1343f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:40.564 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c09e650 0x7f815c138140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54784/0 (socket says 172.21.15.110:54784) 2024-09-15T03:58:40.564 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815affd640 1 -- 172.21.15.110:0/1253851448 learned_addr learned my addr 172.21.15.110:0/1253851448 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:40.564 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815affd640 10 monclient(hunting): get_auth_request con 0x7f815c09e650 auth_method 0 2024-09-15T03:58:40.564 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:40.564 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:40.564 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:40.564 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815b7fe640 10 monclient(hunting): get_auth_request con 0x7f815c0a19d0 auth_method 0 2024-09-15T03:58:40.564 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:40.565 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:40.566 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:40.566 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:40.566 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:40.566 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815a7fc640 1 --2- 172.21.15.110:0/1253851448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f815c09ef60 0x7f815c133eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:40.566 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:40.566 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:40.566 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:40.566 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:40.566 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815a7fc640 10 monclient(hunting): get_auth_request con 0x7f815c09ef60 auth_method 0 2024-09-15T03:58:40.566 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:40.566 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:40.566 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:40.566 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:40.567 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:40.567 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:40.567 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815b7fe640 10 monclient(hunting): handle_auth_done global_id 4188 payload 824 2024-09-15T03:58:40.567 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815b7fe640 1 -- 172.21.15.110:0/1253851448 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c09e650 msgr2=0x7f815c138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:40.567 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815b7fe640 1 --2- 172.21.15.110:0/1253851448 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c09e650 0x7f815c138140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.567 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815b7fe640 1 -- 172.21.15.110:0/1253851448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f815c09ef60 msgr2=0x7f815c133eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:40.567 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.549+0000 7f815b7fe640 1 --2- 172.21.15.110:0/1253851448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f815c09ef60 0x7f815c133eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.567 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.550+0000 7f815b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:58:40.567 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.550+0000 7f815b7fe640 1 monclient: found mon.noname-a 2024-09-15T03:58:40.567 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.550+0000 7f815b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:40.567 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.550+0000 7f815b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:40.567 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.550+0000 7f815b7fe640 1 -- 172.21.15.110:0/1253851448 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f814c009390 con 0x7f815c0a19d0 2024-09-15T03:58:40.567 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.550+0000 7f815affd640 1 --2- 172.21.15.110:0/1253851448 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c09e650 0x7f815c138140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:40.567 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f815b7fe640 1 -- 172.21.15.110:0/1253851448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f815c0a19d0 msgr2=0x7f815c1343f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:40.568 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f815b7fe640 1 --2- 172.21.15.110:0/1253851448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f815c0a19d0 0x7f815c1343f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f814c031000 tx=0x7f814c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:40.568 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f8142ffd640 1 -- 172.21.15.110:0/1253851448 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814c031910 con 0x7f815c0a19d0 2024-09-15T03:58:40.568 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f8142ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:40.568 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f8142ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:40.568 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f8142ffd640 10 monclient: dump: 2024-09-15T03:58:40.568 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-09-15T03:58:40.568 INFO:teuthology.misc.health.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:40.568 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.568 INFO:teuthology.misc.health.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.568 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:40.568 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:40.568 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:40.569 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:40.569 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:40.569 INFO:teuthology.misc.health.smithi110.stderr: 2024-09-15T03:58:40.569 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f8142ffd640 10 monclient: _finish_auth 0 2024-09-15T03:58:40.569 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f8142ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:40.569 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f8142ffd640 1 -- 172.21.15.110:0/1253851448 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f814c031a90 con 0x7f815c0a19d0 2024-09-15T03:58:40.569 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f816191d640 5 monclient: authenticate success, global_id 4188 2024-09-15T03:58:40.569 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f8142ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:40.569 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f816191d640 10 monclient: _renew_subs 2024-09-15T03:58:40.569 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f8142ffd640 1 -- 172.21.15.110:0/1253851448 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814c031e50 con 0x7f815c0a19d0 2024-09-15T03:58:40.569 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f816191d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:40.570 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f816191d640 1 -- 172.21.15.110:0/1253851448 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815c137ab0 con 0x7f815c0a19d0 2024-09-15T03:58:40.570 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f8142ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:40.570 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f8142ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:40.570 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f8142ffd640 10 monclient: dump: 2024-09-15T03:58:40.570 INFO:teuthology.misc.health.smithi110.stderr:epoch 1 2024-09-15T03:58:40.570 INFO:teuthology.misc.health.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:40.570 INFO:teuthology.misc.health.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.570 INFO:teuthology.misc.health.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.570 INFO:teuthology.misc.health.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:40.570 INFO:teuthology.misc.health.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:40.570 INFO:teuthology.misc.health.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:40.570 INFO:teuthology.misc.health.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:40.571 INFO:teuthology.misc.health.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:40.571 INFO:teuthology.misc.health.smithi110.stderr: 2024-09-15T03:58:40.571 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f81597fa640 4 set_mon_vals no callback set 2024-09-15T03:58:40.571 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f816191d640 10 monclient: _renew_subs 2024-09-15T03:58:40.571 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f816191d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:40.571 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.552+0000 7f816191d640 1 -- 172.21.15.110:0/1253851448 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f815c0a47c0 con 0x7f815c0a19d0 2024-09-15T03:58:40.571 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.553+0000 7f8142ffd640 1 -- 172.21.15.110:0/1253851448 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f814c039070 con 0x7f815c0a19d0 2024-09-15T03:58:40.572 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.553+0000 7f816191d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:40.572 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.553+0000 7f816191d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:40.572 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.553+0000 7f816191d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:40.572 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.553+0000 7f816191d640 1 -- 172.21.15.110:0/1253851448 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815c0647e0 con 0x7f815c0a19d0 2024-09-15T03:58:40.572 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.553+0000 7f8142ffd640 1 --2- 172.21.15.110:0/1253851448 >> v2:172.21.15.177:6812/28201 conn(0x7f8148038730 0x7f814803abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:40.572 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.553+0000 7f8142ffd640 1 -- 172.21.15.110:0/1253851448 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3149+0+0 (secure 0 0 0) 0x7f814c06bde0 con 0x7f815c0a19d0 2024-09-15T03:58:40.572 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.553+0000 7f815affd640 1 --2- 172.21.15.110:0/1253851448 >> v2:172.21.15.177:6812/28201 conn(0x7f8148038730 0x7f814803abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:40.572 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.553+0000 7f815affd640 10 monclient: get_auth_request con 0x7f8148038730 auth_method 0 2024-09-15T03:58:40.572 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.556+0000 7f815affd640 1 --2- 172.21.15.110:0/1253851448 >> v2:172.21.15.177:6812/28201 conn(0x7f8148038730 0x7f814803abe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8154002ed0 tx=0x7f8154007ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:40.573 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.558+0000 7f8142ffd640 1 -- 172.21.15.110:0/1253851448 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f814c03c620 con 0x7f815c0a19d0 2024-09-15T03:58:40.573 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.558+0000 7f8142ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:40.573 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.558+0000 7f8142ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:40.714 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.713+0000 7f816191d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:40.714 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.713+0000 7f816191d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:40.715 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.713+0000 7f816191d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:40.715 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.713+0000 7f816191d640 1 -- 172.21.15.110:0/1253851448 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f815c0647e0 con 0x7f815c0a19d0 2024-09-15T03:58:40.715 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.714+0000 7f8142ffd640 1 -- 172.21.15.110:0/1253851448 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+8481 (secure 0 0 0) 0x7f814c040030 con 0x7f815c0a19d0 2024-09-15T03:58:40.716 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.714+0000 7f8142ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:40.716 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.714+0000 7f8142ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:40.716 INFO:teuthology.misc.health.smithi110.stdout: 2024-09-15T03:58:40.716 INFO:teuthology.misc.health.smithi110.stdout:{"epoch":15,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","created":"2024-09-15T03:58:28.265907+0000","modified":"2024-09-15T03:58:39.047818+0000","last_up_change":"2024-09-15T03:58:35.032799+0000","last_in_change":"2024-09-15T03:58:31.538597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:58:36.398831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"105a3b3d-e410-47fb-ac54-6a5d5da4a46f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":28367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":28367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":28367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":28367}]},"public_addr":"172.21.15.110:6808/28367","cluster_addr":"172.21.15.110:6809/28367","heartbeat_back_addr":"172.21.15.110:6811/28367","heartbeat_front_addr":"172.21.15.110:6810/28367","state":["exists","up"]},{"osd":1,"uuid":"77f0f205-6c64-495a-8edd-043de180efa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":28369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":28369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":28369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":28369}]},"public_addr":"172.21.15.110:6800/28369","cluster_addr":"172.21.15.110:6801/28369","heartbeat_back_addr":"172.21.15.110:6803/28369","heartbeat_front_addr":"172.21.15.110:6802/28369","state":["exists","up"]},{"osd":2,"uuid":"e8a42031-303e-4ad0-9896-e5b84cc95cfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":28368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":28368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":28368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":28368}]},"public_addr":"172.21.15.110:6804/28368","cluster_addr":"172.21.15.110:6805/28368","heartbeat_back_addr":"172.21.15.110:6807/28368","heartbeat_front_addr":"172.21.15.110:6806/28368","state":["exists","up"]},{"osd":3,"uuid":"0d65db82-4a0e-4f31-a10a-dcd71dcb269f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":28628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6805","nonce":28628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6807","nonce":28628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":28628}]},"public_addr":"172.21.15.177:6804/28628","cluster_addr":"172.21.15.177:6805/28628","heartbeat_back_addr":"172.21.15.177:6807/28628","heartbeat_front_addr":"172.21.15.177:6806/28628","state":["exists","up"]},{"osd":4,"uuid":"e95e4f77-578d-4d03-bb4d-ee72b43bc227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":28629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6801","nonce":28629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6803","nonce":28629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":28629}]},"public_addr":"172.21.15.177:6800/28629","cluster_addr":"172.21.15.177:6801/28629","heartbeat_back_addr":"172.21.15.177:6803/28629","heartbeat_front_addr":"172.21.15.177:6802/28629","state":["exists","up"]},{"osd":5,"uuid":"f16ffee6-1e25-432c-b6c2-909a72ce24a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":28630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6809","nonce":28630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6811","nonce":28630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":28630}]},"public_addr":"172.21.15.177:6808/28630","cluster_addr":"172.21.15.177:6809/28630","heartbeat_back_addr":"172.21.15.177:6811/28630","heartbeat_front_addr":"172.21.15.177:6810/28630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.117560+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.137736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.130715+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.173663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.130009+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.172505+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:58:40.718 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.716+0000 7f816191d640 1 -- 172.21.15.110:0/1253851448 >> v2:172.21.15.177:6812/28201 conn(0x7f8148038730 msgr2=0x7f814803abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:40.718 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.716+0000 7f816191d640 1 --2- 172.21.15.110:0/1253851448 >> v2:172.21.15.177:6812/28201 conn(0x7f8148038730 0x7f814803abe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8154002ed0 tx=0x7f8154007ed0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.718 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.716+0000 7f816191d640 10 monclient: shutdown 2024-09-15T03:58:40.718 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.716+0000 7f816191d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:40.718 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.716+0000 7f816191d640 1 -- 172.21.15.110:0/1253851448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f815c0a19d0 msgr2=0x7f815c1343f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:40.718 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.716+0000 7f816191d640 1 --2- 172.21.15.110:0/1253851448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f815c0a19d0 0x7f815c1343f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f814c031000 tx=0x7f814c032000 comp rx=0 tx=0).stop 2024-09-15T03:58:40.718 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.716+0000 7f816191d640 1 -- 172.21.15.110:0/1253851448 shutdown_connections 2024-09-15T03:58:40.718 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.716+0000 7f816191d640 1 --2- 172.21.15.110:0/1253851448 >> v2:172.21.15.177:6812/28201 conn(0x7f8148038730 0x7f814803abe0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.718 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.716+0000 7f816191d640 1 --2- 172.21.15.110:0/1253851448 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f815c0a19d0 0x7f815c1343f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.718 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.716+0000 7f816191d640 1 --2- 172.21.15.110:0/1253851448 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f815c09ef60 0x7f815c133eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.718 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.716+0000 7f816191d640 1 --2- 172.21.15.110:0/1253851448 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f815c09e650 0x7f815c138140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.718 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.716+0000 7f816191d640 1 -- 172.21.15.110:0/1253851448 >> 172.21.15.110:0/1253851448 conn(0x7f815c069750 msgr2=0x7f815c0a55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:40.718 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.716+0000 7f816191d640 1 -- 172.21.15.110:0/1253851448 shutdown_connections 2024-09-15T03:58:40.719 INFO:teuthology.misc.health.smithi110.stderr:2024-09-15T03:58:40.716+0000 7f816191d640 1 -- 172.21.15.110:0/1253851448 wait complete. 2024-09-15T03:58:40.726 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-15T03:58:40.726 INFO:tasks.ceph:Creating RBD pool 2024-09-15T03:58:40.726 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-15T03:58:40.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.818+0000 7f152159d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:40.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.818+0000 7f152159d640 10 monclient: build_initial_monmap 2024-09-15T03:58:40.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.818+0000 7f152159d640 10 monclient: monmap: 2024-09-15T03:58:40.819 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:40.819 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:40.820 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:40.819368+0000 2024-09-15T03:58:40.820 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:40.819368+0000 2024-09-15T03:58:40.820 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:40.820 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:40.820 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:40.821 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:40.821 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:40.821 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:40.821 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:40.821 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:40.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.818+0000 7f152159d640 1 Processor -- start 2024-09-15T03:58:40.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 1 -- start start 2024-09-15T03:58:40.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 10 monclient: init 2024-09-15T03:58:40.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:40.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-15T03:58:40.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f151c0a1bf0 0x7f151c131810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:40.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 10 monclient(hunting): picked mon.noname-c con 0x7f151c0a1bf0 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:40.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f151c09e780 0x7f151c09eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:40.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 10 monclient(hunting): picked mon.noname-a con 0x7f151c09e780 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:40.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f151c09f760 0x7f151c09fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:40.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 10 monclient(hunting): picked mon.noname-b con 0x7f151c09f760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:40.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:40.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151c0a0c20 con 0x7f151c09e780 2024-09-15T03:58:40.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:40.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151c09cc90 con 0x7f151c0a1bf0 2024-09-15T03:58:40.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:40.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151c09c9b0 con 0x7f151c09f760 2024-09-15T03:58:40.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:40.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f152159d640 10 monclient(hunting): authenticate will time out at 744.463498s 2024-09-15T03:58:40.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f15127fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f151c09e780 0x7f151c09eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:40.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f151c09f760 0x7f151c09fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:40.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f15127fc640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f151c09e780 0x7f151c09eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56296/0 (socket says 172.21.15.110:56296) 2024-09-15T03:58:40.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f15127fc640 1 -- 172.21.15.110:0/3361281312 learned_addr learned my addr 172.21.15.110:0/3361281312 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:40.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f15127fc640 10 monclient(hunting): get_auth_request con 0x7f151c09e780 auth_method 0 2024-09-15T03:58:40.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f15127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:40.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f15127fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:40.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f15127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:40.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 10 monclient(hunting): get_auth_request con 0x7f151c09f760 auth_method 0 2024-09-15T03:58:40.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:40.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:40.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:40.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f15127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:40.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f15127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:40.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f15127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:40.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:40.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:40.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:40.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151affd640 1 --2- 172.21.15.110:0/3361281312 >> v2:172.21.15.177:3300/0 conn(0x7f151c0a1bf0 0x7f151c131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:40.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151affd640 10 monclient(hunting): get_auth_request con 0x7f151c0a1bf0 auth_method 0 2024-09-15T03:58:40.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:40.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:40.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:40.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:40.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:40.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:40.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 10 monclient(hunting): handle_auth_done global_id 4199 payload 274 2024-09-15T03:58:40.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 1 -- 172.21.15.110:0/3361281312 >> v2:172.21.15.177:3300/0 conn(0x7f151c0a1bf0 msgr2=0x7f151c131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:40.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 1 --2- 172.21.15.110:0/3361281312 >> v2:172.21.15.177:3300/0 conn(0x7f151c0a1bf0 0x7f151c131810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 1 -- 172.21.15.110:0/3361281312 >> v2:172.21.15.110:3300/0 conn(0x7f151c09e780 msgr2=0x7f151c09eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:40.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 1 --2- 172.21.15.110:0/3361281312 >> v2:172.21.15.110:3300/0 conn(0x7f151c09e780 0x7f151c09eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:58:40.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 1 monclient: found mon.noname-b 2024-09-15T03:58:40.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:40.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:40.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f151b7fe640 1 -- 172.21.15.110:0/3361281312 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151c0a1fc0 con 0x7f151c09f760 2024-09-15T03:58:40.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.819+0000 7f15127fc640 1 --2- 172.21.15.110:0/3361281312 >> v2:172.21.15.110:3300/0 conn(0x7f151c09e780 0x7f151c09eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:40.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f151b7fe640 1 -- 172.21.15.110:0/3361281312 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f151c09f760 msgr2=0x7f151c09fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:40.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f151b7fe640 1 --2- 172.21.15.110:0/3361281312 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f151c09f760 0x7f151c09fba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f151c072140 tx=0x7f1514009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dad4d07ca837ac17 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:40.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f151a7fc640 1 -- 172.21.15.110:0/3361281312 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514007760 con 0x7f151c09f760 2024-09-15T03:58:40.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f151a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:40.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f151a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:40.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f151a7fc640 10 monclient: dump: 2024-09-15T03:58:40.838 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:40.838 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:40.839 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.839 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.839 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:40.839 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:40.839 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:40.839 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:40.839 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:40.839 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:40.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f151a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:40.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f151a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:40.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f151a7fc640 1 -- 172.21.15.110:0/3361281312 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151402d050 con 0x7f151c09f760 2024-09-15T03:58:40.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f152159d640 5 monclient: authenticate success, global_id 4199 2024-09-15T03:58:40.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f152159d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:40.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f151a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:40.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f152159d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:40.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f152159d640 4 set_mon_vals no callback set 2024-09-15T03:58:40.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f151a7fc640 1 -- 172.21.15.110:0/3361281312 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514007af0 con 0x7f151c09f760 2024-09-15T03:58:40.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f151a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:40.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f151a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:40.840 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.822+0000 7f151a7fc640 10 monclient: dump: 2024-09-15T03:58:40.840 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:40.840 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:40.840 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.841 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.841 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:40.841 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:40.841 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:40.841 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:40.841 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:40.841 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:40.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.823+0000 7f152159d640 10 monclient: shutdown 2024-09-15T03:58:40.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.823+0000 7f152159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:40.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.823+0000 7f152159d640 1 -- 172.21.15.110:0/3361281312 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f151c09f760 msgr2=0x7f151c09fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:40.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.823+0000 7f152159d640 1 --2- 172.21.15.110:0/3361281312 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f151c09f760 0x7f151c09fba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f151c072140 tx=0x7f1514009780 comp rx=0 tx=0).stop 2024-09-15T03:58:40.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.823+0000 7f152159d640 1 -- 172.21.15.110:0/3361281312 shutdown_connections 2024-09-15T03:58:40.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.823+0000 7f152159d640 1 --2- 172.21.15.110:0/3361281312 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f151c09f760 0x7f151c09fba0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.823+0000 7f152159d640 1 --2- 172.21.15.110:0/3361281312 >> v2:172.21.15.110:3300/0 conn(0x7f151c09e780 0x7f151c09eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.823+0000 7f152159d640 1 --2- 172.21.15.110:0/3361281312 >> v2:172.21.15.177:3300/0 conn(0x7f151c0a1bf0 0x7f151c131810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.823+0000 7f152159d640 1 -- 172.21.15.110:0/3361281312 >> 172.21.15.110:0/3361281312 conn(0x7f151c062dd0 msgr2=0x7f151c0631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:40.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.823+0000 7f152159d640 1 -- 172.21.15.110:0/3361281312 shutdown_connections 2024-09-15T03:58:40.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.823+0000 7f152159d640 1 -- 172.21.15.110:0/3361281312 wait complete. 2024-09-15T03:58:40.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.823+0000 7f152159d640 10 monclient: build_initial_monmap 2024-09-15T03:58:40.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.823+0000 7f152159d640 10 monclient: monmap: 2024-09-15T03:58:40.842 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:40.842 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:40.842 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:40.842 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:40.842 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:40.842 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:40.842 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:40.843 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:40.843 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:40.843 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:40.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 1 Processor -- start 2024-09-15T03:58:40.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 1 -- start start 2024-09-15T03:58:40.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 10 monclient: init 2024-09-15T03:58:40.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:40.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:58:40.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f151c09e780 0x7f151c130700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:40.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 10 monclient(hunting): picked mon.noname-b con 0x7f151c09e780 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:40.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f151c09f760 0x7f151c11b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:40.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 10 monclient(hunting): picked mon.noname-a con 0x7f151c09f760 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:40.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f151c0a1bf0 0x7f151c11b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:40.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 10 monclient(hunting): picked mon.noname-c con 0x7f151c0a1bf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:40.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:40.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c11eb10 con 0x7f151c09f760 2024-09-15T03:58:40.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:40.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c11e830 con 0x7f151c09e780 2024-09-15T03:58:40.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:40.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151c11edf0 con 0x7f151c0a1bf0 2024-09-15T03:58:40.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:40.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.824+0000 7f152159d640 10 monclient(hunting): authenticate will time out at 744.468953s 2024-09-15T03:58:40.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f151c09f760 0x7f151c11b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:40.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f151c0a1bf0 0x7f151c11b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:40.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f151c09f760 0x7f151c11b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56304/0 (socket says 172.21.15.110:56304) 2024-09-15T03:58:40.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 1 -- 172.21.15.110:0/3023279404 learned_addr learned my addr 172.21.15.110:0/3023279404 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:40.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 10 monclient(hunting): get_auth_request con 0x7f151c09f760 auth_method 0 2024-09-15T03:58:40.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:40.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:40.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:40.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151b7fe640 10 monclient(hunting): get_auth_request con 0x7f151c0a1bf0 auth_method 0 2024-09-15T03:58:40.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:40.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:40.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:40.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151affd640 1 --2- 172.21.15.110:0/3023279404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f151c09e780 0x7f151c130700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:40.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:40.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:40.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:40.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:40.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:40.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:40.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151affd640 10 monclient(hunting): get_auth_request con 0x7f151c09e780 auth_method 0 2024-09-15T03:58:40.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:40.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:40.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:40.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:40.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:40.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:40.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 10 monclient(hunting): handle_auth_done global_id 4194 payload 824 2024-09-15T03:58:40.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 1 -- 172.21.15.110:0/3023279404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f151c0a1bf0 msgr2=0x7f151c11b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:40.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 1 --2- 172.21.15.110:0/3023279404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f151c0a1bf0 0x7f151c11b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 1 -- 172.21.15.110:0/3023279404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f151c09e780 msgr2=0x7f151c130700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:40.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 1 --2- 172.21.15.110:0/3023279404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f151c09e780 0x7f151c130700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:40.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 10 monclient: _finish_hunting 0 2024-09-15T03:58:40.846 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 1 monclient: found mon.noname-a 2024-09-15T03:58:40.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:40.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:40.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f15127fc640 1 -- 172.21.15.110:0/3023279404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1504009450 con 0x7f151c09f760 2024-09-15T03:58:40.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.825+0000 7f151b7fe640 1 --2- 172.21.15.110:0/3023279404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f151c0a1bf0 0x7f151c11b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:40.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f15127fc640 1 -- 172.21.15.110:0/3023279404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f151c09f760 msgr2=0x7f151c11b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:40.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f15127fc640 1 --2- 172.21.15.110:0/3023279404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f151c09f760 0x7f151c11b3d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f150400ec80 tx=0x7f15040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:40.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f15137fe640 1 -- 172.21.15.110:0/3023279404 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1504009730 con 0x7f151c09f760 2024-09-15T03:58:40.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f15137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:40.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f15137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:40.847 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f15137fe640 10 monclient: dump: 2024-09-15T03:58:40.847 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:40.847 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:40.847 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.848 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.848 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:40.848 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:40.848 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:40.848 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:40.848 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:40.848 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:40.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f15137fe640 10 monclient: _finish_auth 0 2024-09-15T03:58:40.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f15137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:40.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f152159d640 5 monclient: authenticate success, global_id 4194 2024-09-15T03:58:40.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f15137fe640 1 -- 172.21.15.110:0/3023279404 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150400eec0 con 0x7f151c09f760 2024-09-15T03:58:40.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f152159d640 10 monclient: _renew_subs 2024-09-15T03:58:40.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f152159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:40.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f152159d640 1 -- 172.21.15.110:0/3023279404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1514009a20 con 0x7f151c09f760 2024-09-15T03:58:40.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f15137fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:40.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f15137fe640 1 -- 172.21.15.110:0/3023279404 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1504013470 con 0x7f151c09f760 2024-09-15T03:58:40.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f15137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:40.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f15137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:40.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f15137fe640 10 monclient: dump: 2024-09-15T03:58:40.849 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:40.849 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:40.849 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.849 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:40.849 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:40.849 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:40.849 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:40.850 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:40.850 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:40.850 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:40.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f152159d640 10 monclient: _renew_subs 2024-09-15T03:58:40.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f1519ffb640 4 set_mon_vals no callback set 2024-09-15T03:58:40.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f152159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:40.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.827+0000 7f152159d640 1 -- 172.21.15.110:0/3023279404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151c11d8d0 con 0x7f151c09f760 2024-09-15T03:58:40.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.828+0000 7f15137fe640 1 -- 172.21.15.110:0/3023279404 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f1504013820 con 0x7f151c09f760 2024-09-15T03:58:40.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.828+0000 7f15137fe640 1 --2- 172.21.15.110:0/3023279404 >> v2:172.21.15.177:6812/28201 conn(0x7f14fc038700 0x7f14fc03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:40.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.828+0000 7f15137fe640 1 -- 172.21.15.110:0/3023279404 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3149+0+0 (secure 0 0 0) 0x7f150404b890 con 0x7f151c09f760 2024-09-15T03:58:40.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.828+0000 7f152159d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:40.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.828+0000 7f152159d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:40.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.828+0000 7f152159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:40.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.828+0000 7f152159d640 1 -- 172.21.15.110:0/3023279404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151c11efb0 con 0x7f151c09f760 2024-09-15T03:58:40.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.828+0000 7f151affd640 1 --2- 172.21.15.110:0/3023279404 >> v2:172.21.15.177:6812/28201 conn(0x7f14fc038700 0x7f14fc03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:40.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.828+0000 7f151affd640 10 monclient: get_auth_request con 0x7f14fc038700 auth_method 0 2024-09-15T03:58:40.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.831+0000 7f151affd640 1 --2- 172.21.15.110:0/3023279404 >> v2:172.21.15.177:6812/28201 conn(0x7f14fc038700 0x7f14fc03abb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f150c009d20 tx=0x7f150c0090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:40.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.833+0000 7f15137fe640 1 -- 172.21.15.110:0/3023279404 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1504021020 con 0x7f151c09f760 2024-09-15T03:58:40.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.833+0000 7f15137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:40.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.833+0000 7f15137fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:40.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.986+0000 7f152159d640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T03:58:40.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.986+0000 7f152159d640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T03:58:40.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.986+0000 7f152159d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:40.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:40.986+0000 7f152159d640 1 -- 172.21.15.110:0/3023279404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f151c070950 con 0x7f151c09f760 2024-09-15T03:58:41.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.059+0000 7f15137fe640 1 -- 172.21.15.110:0/3023279404 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) v1 ==== 109+0+0 (secure 0 0 0) 0x7f15040236b0 con 0x7f151c09f760 2024-09-15T03:58:41.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.059+0000 7f15137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T03:58:41.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.059+0000 7f15137fe640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-15T03:58:41.060 INFO:teuthology.orchestra.run.smithi110.stderr:pool 'rbd' created 2024-09-15T03:58:41.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.061+0000 7f152159d640 1 -- 172.21.15.110:0/3023279404 >> v2:172.21.15.177:6812/28201 conn(0x7f14fc038700 msgr2=0x7f14fc03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:41.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.061+0000 7f152159d640 1 --2- 172.21.15.110:0/3023279404 >> v2:172.21.15.177:6812/28201 conn(0x7f14fc038700 0x7f14fc03abb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f150c009d20 tx=0x7f150c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:41.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.061+0000 7f152159d640 10 monclient: shutdown 2024-09-15T03:58:41.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.061+0000 7f152159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:41.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.061+0000 7f152159d640 1 -- 172.21.15.110:0/3023279404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f151c09f760 msgr2=0x7f151c11b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:41.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.061+0000 7f152159d640 1 --2- 172.21.15.110:0/3023279404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f151c09f760 0x7f151c11b3d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f150400ec80 tx=0x7f15040091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:41.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.061+0000 7f152159d640 1 -- 172.21.15.110:0/3023279404 shutdown_connections 2024-09-15T03:58:41.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.061+0000 7f152159d640 1 --2- 172.21.15.110:0/3023279404 >> v2:172.21.15.177:6812/28201 conn(0x7f14fc038700 0x7f14fc03abb0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:41.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.061+0000 7f152159d640 1 --2- 172.21.15.110:0/3023279404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f151c0a1bf0 0x7f151c11b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:41.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.061+0000 7f152159d640 1 --2- 172.21.15.110:0/3023279404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f151c09f760 0x7f151c11b3d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:41.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.061+0000 7f152159d640 1 --2- 172.21.15.110:0/3023279404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f151c09e780 0x7f151c130700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:41.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.061+0000 7f152159d640 1 -- 172.21.15.110:0/3023279404 >> 172.21.15.110:0/3023279404 conn(0x7f151c062dd0 msgr2=0x7f151c0a2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:41.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.061+0000 7f152159d640 1 -- 172.21.15.110:0/3023279404 shutdown_connections 2024-09-15T03:58:41.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.062+0000 7f152159d640 1 -- 172.21.15.110:0/3023279404 wait complete. 2024-09-15T03:58:41.072 DEBUG:teuthology.orchestra.run.smithi110:> rbd --cluster ceph pool init rbd 2024-09-15T03:58:41.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.123+0000 7f799ccd1b80 10 monclient: get_monmap_and_config 2024-09-15T03:58:41.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.123+0000 7f799ccd1b80 10 monclient: build_initial_monmap 2024-09-15T03:58:41.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.123+0000 7f799ccd1b80 10 monclient: monmap: 2024-09-15T03:58:41.124 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:41.124 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:41.124 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:41.124087+0000 2024-09-15T03:58:41.124 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:41.124087+0000 2024-09-15T03:58:41.124 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:41.125 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:41.125 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:41.125 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:41.125 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:41.125 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:41.125 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:41.125 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:41.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.123+0000 7f799ccd1b80 1 Processor -- start 2024-09-15T03:58:41.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.123+0000 7f799ccd1b80 1 -- start start 2024-09-15T03:58:41.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.123+0000 7f799ccd1b80 10 monclient: init 2024-09-15T03:58:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799ccd1b80 10 monclient: _reopen_session rank -1 2024-09-15T03:58:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799ccd1b80 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-15T03:58:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799ccd1b80 10 monclient(hunting): picked mon.noname-a-legacy con 0x55a8e337e470 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799ccd1b80 10 monclient(hunting): picked mon.noname-c-legacy con 0x55a8e337dbd0 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799ccd1b80 1 --2- >> v2:172.21.15.110:3300/0 conn(0x55a8e34165d0 0x55a8e34189b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799ccd1b80 10 monclient(hunting): picked mon.noname-a con 0x55a8e34165d0 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799ccd1b80 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55a8e329f0e0 con 0x55a8e337e470 2024-09-15T03:58:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799ccd1b80 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55a8e3278270 con 0x55a8e337dbd0 2024-09-15T03:58:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799ccd1b80 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799ccd1b80 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x55a8e32d8260 con 0x55a8e34165d0 2024-09-15T03:58:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799ccd1b80 10 monclient(hunting): _renew_subs 2024-09-15T03:58:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799ccd1b80 10 monclient(hunting): authenticate will time out at 744.768271s 2024-09-15T03:58:41.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x55a8e34165d0 0x55a8e34189b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:41.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f7999a4b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x55a8e337e470 0x55a8e337e840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33708/0 (socket says 172.21.15.110:33708) 2024-09-15T03:58:41.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f7999a4b640 1 -- 172.21.15.110:0/3162575997 learned_addr learned my addr 172.21.15.110:0/3162575997 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:41.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 10 monclient(hunting): get_auth_request con 0x55a8e34165d0 auth_method 0 2024-09-15T03:58:41.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:41.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:41.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:41.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:41.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:41.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:41.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 10 monclient(hunting): handle_auth_done global_id 4197 payload 274 2024-09-15T03:58:41.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 1 -- 172.21.15.110:0/3162575997 >> v1:172.21.15.177:6789/0 conn(0x55a8e337dbd0 legacy=0x55a8e3414eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:41.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 1 -- 172.21.15.110:0/3162575997 >> v1:172.21.15.110:6789/0 conn(0x55a8e337e470 legacy=0x55a8e337e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:41.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 10 monclient: _finish_hunting 0 2024-09-15T03:58:41.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 1 monclient: found mon.noname-a 2024-09-15T03:58:41.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:41.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:41.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.124+0000 7f799a24c640 1 -- 172.21.15.110:0/3162575997 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8e32850e0 con 0x55a8e34165d0 2024-09-15T03:58:41.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f799a24c640 1 -- 172.21.15.110:0/3162575997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8e34165d0 msgr2=0x55a8e34189b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:41.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f799a24c640 1 --2- 172.21.15.110:0/3162575997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8e34165d0 0x55a8e34189b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f798c00a5a0 tx=0x7f798c045bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e77d77b821c167 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:41.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f7998a49640 1 -- 172.21.15.110:0/3162575997 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c058480 con 0x55a8e34165d0 2024-09-15T03:58:41.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f7998a49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:41.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f7998a49640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:41.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f7998a49640 10 monclient: dump: 2024-09-15T03:58:41.136 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:41.136 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:41.136 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:41.136 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:41.136 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:41.136 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:41.136 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:41.137 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:41.137 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:41.137 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:41.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f7998a49640 10 monclient: _finish_auth 0 2024-09-15T03:58:41.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f7998a49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:41.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f7998a49640 1 -- 172.21.15.110:0/3162575997 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798c0585e0 con 0x55a8e34165d0 2024-09-15T03:58:41.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f799ccd1b80 5 monclient: authenticate success, global_id 4197 2024-09-15T03:58:41.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f799ccd1b80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:41.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f7998a49640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:41.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f799ccd1b80 10 monclient: get_monmap_and_config success 2024-09-15T03:58:41.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f7998a49640 1 -- 172.21.15.110:0/3162575997 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c058790 con 0x55a8e34165d0 2024-09-15T03:58:41.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f799ccd1b80 4 set_mon_vals no callback set 2024-09-15T03:58:41.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f7998a49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:41.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f7998a49640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:41.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.128+0000 7f7998a49640 10 monclient: dump: 2024-09-15T03:58:41.138 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:41.138 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:41.138 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:41.138 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:41.138 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:41.138 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:41.138 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:41.138 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:41.138 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:41.139 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:41.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.129+0000 7f799ccd1b80 10 monclient: shutdown 2024-09-15T03:58:41.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.129+0000 7f799ccd1b80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:41.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.129+0000 7f799ccd1b80 1 -- 172.21.15.110:0/3162575997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8e34165d0 msgr2=0x55a8e34189b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:41.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.129+0000 7f799ccd1b80 1 --2- 172.21.15.110:0/3162575997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8e34165d0 0x55a8e34189b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f798c00a5a0 tx=0x7f798c045bd0 comp rx=0 tx=0).stop 2024-09-15T03:58:41.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.129+0000 7f799ccd1b80 1 -- 172.21.15.110:0/3162575997 shutdown_connections 2024-09-15T03:58:41.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.129+0000 7f799ccd1b80 1 --2- 172.21.15.110:0/3162575997 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8e34165d0 0x55a8e34189b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:41.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.129+0000 7f799ccd1b80 1 -- 172.21.15.110:0/3162575997 >> 172.21.15.110:0/3162575997 conn(0x55a8e3380090 msgr2=0x55a8e3380490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:41.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.129+0000 7f799ccd1b80 1 -- 172.21.15.110:0/3162575997 shutdown_connections 2024-09-15T03:58:41.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.129+0000 7f799ccd1b80 1 -- 172.21.15.110:0/3162575997 wait complete. 2024-09-15T03:58:41.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.129+0000 7f799ccd1b80 10 monclient: build_initial_monmap 2024-09-15T03:58:41.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.129+0000 7f799ccd1b80 10 monclient: monmap: 2024-09-15T03:58:41.139 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:41.140 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:41.140 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:41.140 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:41.140 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:41.140 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:41.140 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:41.140 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:41.140 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:41.140 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:41.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.129+0000 7f799ccd1b80 1 Processor -- start 2024-09-15T03:58:41.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 1 -- start start 2024-09-15T03:58:41.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 10 monclient: init 2024-09-15T03:58:41.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 10 monclient: _reopen_session rank -1 2024-09-15T03:58:41.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:58:41.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55a8e337dbd0 0x55a8e34123e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:41.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 10 monclient(hunting): picked mon.noname-c con 0x55a8e337dbd0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:41.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8e337e470 0x55a8e33754a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:41.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 10 monclient(hunting): picked mon.noname-a con 0x55a8e337e470 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:41.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8e34165d0 0x55a8e33759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:41.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 10 monclient(hunting): picked mon.noname-b con 0x55a8e34165d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:41.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:41.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8e32d3f40 con 0x55a8e337e470 2024-09-15T03:58:41.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:41.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8e3296ba0 con 0x55a8e34165d0 2024-09-15T03:58:41.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:41.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a8e3296840 con 0x55a8e337dbd0 2024-09-15T03:58:41.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 10 monclient(hunting): _renew_subs 2024-09-15T03:58:41.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799ccd1b80 10 monclient(hunting): authenticate will time out at 744.774451s 2024-09-15T03:58:41.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8e337e470 0x55a8e33754a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:41.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f7999a4b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55a8e337dbd0 0x55a8e34123e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:41.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8e337e470 0x55a8e33754a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56326/0 (socket says 172.21.15.110:56326) 2024-09-15T03:58:41.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 1 -- 172.21.15.110:0/1863596818 learned_addr learned my addr 172.21.15.110:0/1863596818 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:41.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 10 monclient(hunting): get_auth_request con 0x55a8e337e470 auth_method 0 2024-09-15T03:58:41.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:41.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:41.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:41.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f7999a4b640 10 monclient(hunting): get_auth_request con 0x55a8e337dbd0 auth_method 0 2024-09-15T03:58:41.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f7999a4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:41.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f7999a4b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:41.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f7999a4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:41.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:41.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:41.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:41.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f7999a4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:41.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f7999a4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:41.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f7999a4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:41.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799a24c640 1 --2- 172.21.15.110:0/1863596818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8e34165d0 0x55a8e33759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:41.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799a24c640 10 monclient(hunting): get_auth_request con 0x55a8e34165d0 auth_method 0 2024-09-15T03:58:41.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799a24c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:41.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799a24c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:41.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799a24c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:41.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799a24c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:41.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799a24c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:41.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799a24c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:41.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 10 monclient(hunting): handle_auth_done global_id 4200 payload 824 2024-09-15T03:58:41.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 1 -- 172.21.15.110:0/1863596818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55a8e337dbd0 msgr2=0x55a8e34123e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:41.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 1 --2- 172.21.15.110:0/1863596818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55a8e337dbd0 0x55a8e34123e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:41.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 1 -- 172.21.15.110:0/1863596818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8e34165d0 msgr2=0x55a8e33759e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:41.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 1 --2- 172.21.15.110:0/1863596818 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x55a8e34165d0 0x55a8e33759e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:41.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 10 monclient: _finish_hunting 0 2024-09-15T03:58:41.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 1 monclient: found mon.noname-a 2024-09-15T03:58:41.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:41.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:41.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f799924a640 1 -- 172.21.15.110:0/1863596818 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c00a0e0 con 0x55a8e337e470 2024-09-15T03:58:41.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.130+0000 7f7999a4b640 1 --2- 172.21.15.110:0/1863596818 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x55a8e337dbd0 0x55a8e34123e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:41.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f799924a640 1 -- 172.21.15.110:0/1863596818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8e337e470 msgr2=0x55a8e33754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:41.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f799924a640 1 --2- 172.21.15.110:0/1863596818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8e337e470 0x55a8e33754a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7984002c20 tx=0x7f79840028b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:41.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f7989ffb640 1 -- 172.21.15.110:0/1863596818 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7984002d80 con 0x55a8e337e470 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f7989ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f7989ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f7989ffb640 10 monclient: dump: 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f7989ffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:41.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f7989ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f7989ffb640 1 -- 172.21.15.110:0/1863596818 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7984008030 con 0x55a8e337e470 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f799ccd1b80 5 monclient: authenticate success, global_id 4200 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f799ccd1b80 10 monclient: _renew_subs 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f799ccd1b80 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f799ccd1b80 1 -- 172.21.15.110:0/1863596818 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a8e3278600 con 0x55a8e337e470 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f7989ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f7989ffb640 1 -- 172.21.15.110:0/1863596818 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798400f650 con 0x55a8e337e470 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f7989ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f7989ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f7989ffb640 10 monclient: dump: 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:41.146 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:41.147 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:41.147 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:41.147 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:41.147 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:41.147 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:41.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f798bfff640 4 set_mon_vals no callback set 2024-09-15T03:58:41.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f799ccd1b80 10 monclient: _renew_subs 2024-09-15T03:58:41.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f799ccd1b80 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:41.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.133+0000 7f799ccd1b80 1 -- 172.21.15.110:0/1863596818 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a8e3278060 con 0x55a8e337e470 2024-09-15T03:58:41.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.134+0000 7f7989ffb640 1 -- 172.21.15.110:0/1863596818 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f798400fac0 con 0x55a8e337e470 2024-09-15T03:58:41.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.134+0000 7f7989ffb640 1 --2- 172.21.15.110:0/1863596818 >> v2:172.21.15.177:6812/28201 conn(0x7f7978037030 0x7f79780394e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:41.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.134+0000 7f7989ffb640 1 -- 172.21.15.110:0/1863596818 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3491+0+0 (secure 0 0 0) 0x7f7984049ce0 con 0x55a8e337e470 2024-09-15T03:58:41.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.134+0000 7f799ccd1b80 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T03:58:41.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.134+0000 7f799ccd1b80 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T03:58:41.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.134+0000 7f799ccd1b80 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:41.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.134+0000 7f799ccd1b80 1 -- 172.21.15.110:0/1863596818 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) v1 -- 0x55a8e3411980 con 0x55a8e337e470 2024-09-15T03:58:41.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.134+0000 7f7999a4b640 1 --2- 172.21.15.110:0/1863596818 >> v2:172.21.15.177:6812/28201 conn(0x7f7978037030 0x7f79780394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:41.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.134+0000 7f7999a4b640 10 monclient: get_auth_request con 0x7f7978037030 auth_method 0 2024-09-15T03:58:41.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:41.137+0000 7f7999a4b640 1 --2- 172.21.15.110:0/1863596818 >> v2:172.21.15.177:6812/28201 conn(0x7f7978037030 0x7f79780394e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f799000cb10 tx=0x7f79900068c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:42.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.065+0000 7f7989ffb640 1 -- 172.21.15.110:0/1863596818 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v17) v1 ==== 141+0+0 (secure 0 0 0) 0x7f7984018450 con 0x55a8e337e470 2024-09-15T03:58:42.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.065+0000 7f7989ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T03:58:42.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.065+0000 7f7989ffb640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-15T03:58:42.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.065+0000 7f799ccd1b80 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:42.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.065+0000 7f799ccd1b80 1 -- 172.21.15.110:0/1863596818 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55a8e3296630 con 0x55a8e337e470 2024-09-15T03:58:42.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.066+0000 7f7989ffb640 1 -- 172.21.15.110:0/1863596818 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79840029f0 con 0x55a8e337e470 2024-09-15T03:58:42.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.066+0000 7f7989ffb640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-15T03:58:42.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.066+0000 7f798bfff640 10 monclient: _renew_subs 2024-09-15T03:58:42.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.066+0000 7f798bfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:42.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.066+0000 7f798bfff640 1 -- 172.21.15.110:0/1863596818 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=17}) v3 -- 0x7f7974002990 con 0x55a8e337e470 2024-09-15T03:58:42.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.066+0000 7f7989ffb640 1 -- 172.21.15.110:0/1863596818 <== mon.0 v2:172.21.15.110:3300/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 650+0+0 (secure 0 0 0) 0x7f7984016d50 con 0x55a8e337e470 2024-09-15T03:58:42.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.066+0000 7f799ccd1b80 1 --2- 172.21.15.110:0/1863596818 >> v2:172.21.15.177:6808/28630 conn(0x55a8e3424f60 0x55a8e3427340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:42.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.066+0000 7f799ccd1b80 1 -- 172.21.15.110:0/1863596818 --> v2:172.21.15.177:6808/28630 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) v8 -- 0x55a8e3427880 con 0x55a8e3424f60 2024-09-15T03:58:42.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.066+0000 7f799a24c640 1 --2- 172.21.15.110:0/1863596818 >> v2:172.21.15.177:6808/28630 conn(0x55a8e3424f60 0x55a8e3427340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:42.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.066+0000 7f799a24c640 10 monclient: get_auth_request con 0x55a8e3424f60 auth_method 0 2024-09-15T03:58:42.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.071+0000 7f799a24c640 1 --2- 172.21.15.110:0/1863596818 >> v2:172.21.15.177:6808/28630 conn(0x55a8e3424f60 0x55a8e3427340 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:42.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.072+0000 7f799a24c640 1 -- 172.21.15.110:0/1863596818 <== osd.5 v2:172.21.15.177:6808/28630 1 ==== osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) v8 ==== 153+0+0 (crc 0 0 0) 0x7f798c058880 con 0x55a8e3424f60 2024-09-15T03:58:42.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.074+0000 7f799ccd1b80 1 -- 172.21.15.110:0/1863596818 --> v2:172.21.15.177:6808/28630 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) v8 -- 0x55a8e3474d10 con 0x55a8e3424f60 2024-09-15T03:58:42.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.074+0000 7f799a24c640 1 -- 172.21.15.110:0/1863596818 <== osd.5 v2:172.21.15.177:6808/28630 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f798c058880 con 0x55a8e3424f60 2024-09-15T03:58:42.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.075+0000 7f799ccd1b80 1 -- 172.21.15.110:0/1863596818 --> v2:172.21.15.177:6808/28630 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) v8 -- 0x55a8e3473df0 con 0x55a8e3424f60 2024-09-15T03:58:42.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.075+0000 7f799a24c640 1 -- 172.21.15.110:0/1863596818 <== osd.5 v2:172.21.15.177:6808/28630 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (crc 0 0 0) 0x7f798c058880 con 0x55a8e3424f60 2024-09-15T03:58:42.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.075+0000 7f798b7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:42.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:42.075+0000 7f798b7fe640 1 -- 172.21.15.110:0/1863596818 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) v4 -- 0x7f796c001ee0 con 0x55a8e337e470 2024-09-15T03:58:43.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:43.070+0000 7f7989ffb640 1 -- 172.21.15.110:0/1863596818 <== mon.0 v2:172.21.15.110:3300/0 9 ==== pool_op_reply(tid 4 (0) Success v18) v1 ==== 55+0+0 (secure 0 0 0) 0x7f7984016360 con 0x55a8e337e470 2024-09-15T03:58:43.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:43.070+0000 7f7989ffb640 10 monclient: _renew_subs 2024-09-15T03:58:43.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:43.070+0000 7f7989ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:43.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:43.070+0000 7f7989ffb640 1 -- 172.21.15.110:0/1863596818 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=18}) v3 -- 0x7f797803ced0 con 0x55a8e337e470 2024-09-15T03:58:43.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:43.070+0000 7f7989ffb640 1 -- 172.21.15.110:0/1863596818 <== mon.0 v2:172.21.15.110:3300/0 10 ==== osd_map(18..18 src has 1..18) v4 ==== 618+0+0 (secure 0 0 0) 0x7f798400fd70 con 0x55a8e337e470 2024-09-15T03:58:43.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:43.070+0000 7f798bfff640 1 -- 172.21.15.110:0/1863596818 --> v2:172.21.15.177:6808/28630 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) v8 -- 0x7f7974005450 con 0x55a8e3424f60 2024-09-15T03:58:43.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:43.073+0000 7f799a24c640 1 -- 172.21.15.110:0/1863596818 <== osd.5 v2:172.21.15.177:6808/28630 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) v8 ==== 194+0+0 (crc 0 0 0) 0x7f798c058880 con 0x55a8e3424f60 2024-09-15T03:58:43.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:43.073+0000 7f798b7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:43.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:43.073+0000 7f798b7fe640 1 -- 172.21.15.110:0/1863596818 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v18) v4 -- 0x7f796c0033b0 con 0x55a8e337e470 2024-09-15T03:58:44.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.071+0000 7f7989ffb640 1 -- 172.21.15.110:0/1863596818 <== mon.0 v2:172.21.15.110:3300/0 11 ==== pool_op_reply(tid 6 (0) Success v19) v1 ==== 43+0+0 (secure 0 0 0) 0x7f79840169b0 con 0x55a8e337e470 2024-09-15T03:58:44.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.071+0000 7f7989ffb640 10 monclient: _renew_subs 2024-09-15T03:58:44.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.071+0000 7f7989ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:44.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.071+0000 7f7989ffb640 1 -- 172.21.15.110:0/1863596818 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=19}) v3 -- 0x7f797803d060 con 0x55a8e337e470 2024-09-15T03:58:44.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.072+0000 7f7989ffb640 1 -- 172.21.15.110:0/1863596818 <== mon.0 v2:172.21.15.110:3300/0 12 ==== osd_map(19..19 src has 1..19) v4 ==== 646+0+0 (secure 0 0 0) 0x7f7984048bd0 con 0x55a8e337e470 2024-09-15T03:58:44.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.072+0000 7f798bfff640 1 -- 172.21.15.110:0/1863596818 --> v2:172.21.15.177:6808/28630 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) v8 -- 0x7f7974005820 con 0x55a8e3424f60 2024-09-15T03:58:44.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.074+0000 7f799a24c640 1 -- 172.21.15.110:0/1863596818 <== osd.5 v2:172.21.15.177:6808/28630 5 ==== osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) v8 ==== 152+0+0 (crc 0 0 0) 0x7f798c058880 con 0x55a8e3424f60 2024-09-15T03:58:44.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.074+0000 7f799ccd1b80 1 -- 172.21.15.110:0/1863596818 >> v2:172.21.15.177:6808/28630 conn(0x55a8e3424f60 msgr2=0x55a8e3427340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.074+0000 7f799ccd1b80 1 --2- 172.21.15.110:0/1863596818 >> v2:172.21.15.177:6808/28630 conn(0x55a8e3424f60 0x55a8e3427340 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.074+0000 7f799ccd1b80 1 -- 172.21.15.110:0/1863596818 >> v2:172.21.15.177:6812/28201 conn(0x7f7978037030 msgr2=0x7f79780394e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.074+0000 7f799ccd1b80 1 --2- 172.21.15.110:0/1863596818 >> v2:172.21.15.177:6812/28201 conn(0x7f7978037030 0x7f79780394e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f799000cb10 tx=0x7f79900068c0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.074+0000 7f799ccd1b80 10 monclient: shutdown 2024-09-15T03:58:44.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.074+0000 7f799ccd1b80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:44.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.074+0000 7f799ccd1b80 1 -- 172.21.15.110:0/1863596818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8e337e470 msgr2=0x55a8e33754a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.074+0000 7f799ccd1b80 1 --2- 172.21.15.110:0/1863596818 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x55a8e337e470 0x55a8e33754a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7984002c20 tx=0x7f79840028b0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.074+0000 7f799a24c640 1 -- 172.21.15.110:0/1863596818 reap_dead start 2024-09-15T03:58:44.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.074+0000 7f799ccd1b80 1 -- 172.21.15.110:0/1863596818 shutdown_connections 2024-09-15T03:58:44.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.074+0000 7f799ccd1b80 1 -- 172.21.15.110:0/1863596818 >> 172.21.15.110:0/1863596818 conn(0x55a8e3380090 msgr2=0x55a8e3381970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:44.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.074+0000 7f799ccd1b80 1 -- 172.21.15.110:0/1863596818 shutdown_connections 2024-09-15T03:58:44.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.074+0000 7f799ccd1b80 1 -- 172.21.15.110:0/1863596818 wait complete. 2024-09-15T03:58:44.080 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-15T03:58:44.081 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T03:58:44.081 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-15T03:58:44.081 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-15T03:58:44.081 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-15T03:58:44.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.179+0000 7f9e5f91d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.179+0000 7f9e5f91d640 10 monclient: build_initial_monmap 2024-09-15T03:58:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.179+0000 7f9e5f91d640 10 monclient: monmap: 2024-09-15T03:58:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:44.180962+0000 2024-09-15T03:58:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:44.180962+0000 2024-09-15T03:58:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:44.181 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:44.182 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.180+0000 7f9e5f91d640 1 Processor -- start 2024-09-15T03:58:44.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.180+0000 7f9e5f91d640 1 -- start start 2024-09-15T03:58:44.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.180+0000 7f9e5f91d640 10 monclient: init 2024-09-15T03:58:44.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5f91d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:44.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5f91d640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-15T03:58:44.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5f91d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e5806e430 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:44.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5f91d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9e5806e800 0x7f9e5806ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5f91d640 10 monclient(hunting): picked mon.noname-a con 0x7f9e5806e800 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:44.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5f91d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e58133610 0x7f9e581359f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5f91d640 10 monclient(hunting): picked mon.noname-b con 0x7f9e58133610 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:44.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5f91d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5809b660 con 0x7f9e5806e430 2024-09-15T03:58:44.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5f91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5f91d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e5806c940 con 0x7f9e5806e800 2024-09-15T03:58:44.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5f91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5f91d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5806b0f0 con 0x7f9e58133610 2024-09-15T03:58:44.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5f91d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:44.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5f91d640 10 monclient(hunting): authenticate will time out at 747.825445s 2024-09-15T03:58:44.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9e5806e800 0x7f9e5806ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5deba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e58133610 0x7f9e581359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5d6b9640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9e5806e430 0x7f9e5806a9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33712/0 (socket says 172.21.15.110:33712) 2024-09-15T03:58:44.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5d6b9640 1 -- 172.21.15.110:0/22182646 learned_addr learned my addr 172.21.15.110:0/22182646 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:44.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5deba640 10 monclient(hunting): get_auth_request con 0x7f9e58133610 auth_method 0 2024-09-15T03:58:44.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5deba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:44.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5deba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:44.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5deba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:44.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 10 monclient(hunting): get_auth_request con 0x7f9e5806e800 auth_method 0 2024-09-15T03:58:44.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:44.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:44.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:44.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:44.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:44.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:44.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5deba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:44.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5deba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:44.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5deba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:44.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 10 monclient(hunting): handle_auth_done global_id 4203 payload 274 2024-09-15T03:58:44.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 1 -- 172.21.15.110:0/22182646 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e58133610 msgr2=0x7f9e581359f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:44.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 1 --2- 172.21.15.110:0/22182646 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e58133610 0x7f9e581359f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 1 -- 172.21.15.110:0/22182646 >> v1:172.21.15.110:6789/0 conn(0x7f9e5806e430 legacy=0x7f9e5806a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:44.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 10 monclient: _finish_hunting 0 2024-09-15T03:58:44.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 1 monclient: found mon.noname-a 2024-09-15T03:58:44.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:44.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:44.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5ceb8640 1 -- 172.21.15.110:0/22182646 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5806cc20 con 0x7f9e5806e800 2024-09-15T03:58:44.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.181+0000 7f9e5deba640 1 --2- 172.21.15.110:0/22182646 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e58133610 0x7f9e581359f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:44.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e5ceb8640 1 -- 172.21.15.110:0/22182646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e5806e800 msgr2=0x7f9e5806ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:44.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e5ceb8640 1 --2- 172.21.15.110:0/22182646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e5806e800 0x7f9e5806ec20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009850 tx=0x7f9e480090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2d473b7c8cd5dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:44.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e4ffff640 1 -- 172.21.15.110:0/22182646 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4802adf0 con 0x7f9e5806e800 2024-09-15T03:58:44.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:44.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e4ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:44.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e4ffff640 10 monclient: dump: 2024-09-15T03:58:44.193 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:44.193 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.193 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.193 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.193 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:44.193 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.193 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:44.193 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:44.193 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:44.193 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e4ffff640 10 monclient: _finish_auth 0 2024-09-15T03:58:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e4ffff640 1 -- 172.21.15.110:0/22182646 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e48032440 con 0x7f9e5806e800 2024-09-15T03:58:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e4ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e4ffff640 1 -- 172.21.15.110:0/22182646 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e480327e0 con 0x7f9e5806e800 2024-09-15T03:58:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e5f91d640 5 monclient: authenticate success, global_id 4203 2024-09-15T03:58:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e5f91d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e5f91d640 4 set_mon_vals no callback set 2024-09-15T03:58:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e4ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.184+0000 7f9e4ffff640 10 monclient: dump: 2024-09-15T03:58:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.194 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:44.195 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.185+0000 7f9e5f91d640 10 monclient: shutdown 2024-09-15T03:58:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.185+0000 7f9e5f91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.185+0000 7f9e5f91d640 1 -- 172.21.15.110:0/22182646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e5806e800 msgr2=0x7f9e5806ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.185+0000 7f9e5f91d640 1 --2- 172.21.15.110:0/22182646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e5806e800 0x7f9e5806ec20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009850 tx=0x7f9e480090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.185+0000 7f9e5f91d640 1 -- 172.21.15.110:0/22182646 shutdown_connections 2024-09-15T03:58:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.185+0000 7f9e5f91d640 1 --2- 172.21.15.110:0/22182646 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e58133610 0x7f9e581359f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.185+0000 7f9e5f91d640 1 --2- 172.21.15.110:0/22182646 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e5806e800 0x7f9e5806ec20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.185+0000 7f9e5f91d640 1 -- 172.21.15.110:0/22182646 >> 172.21.15.110:0/22182646 conn(0x7f9e5809d8f0 msgr2=0x7f9e5809dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.185+0000 7f9e5f91d640 1 -- 172.21.15.110:0/22182646 shutdown_connections 2024-09-15T03:58:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.185+0000 7f9e5f91d640 1 -- 172.21.15.110:0/22182646 wait complete. 2024-09-15T03:58:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.185+0000 7f9e5f91d640 10 monclient: build_initial_monmap 2024-09-15T03:58:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.185+0000 7f9e5f91d640 10 monclient: monmap: 2024-09-15T03:58:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:44.196 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:44.197 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 1 Processor -- start 2024-09-15T03:58:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 1 -- start start 2024-09-15T03:58:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 10 monclient: init 2024-09-15T03:58:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:58:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e5806e430 0x7f9e5813f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 10 monclient(hunting): picked mon.noname-c con 0x7f9e5806e430 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e5806e800 0x7f9e5813f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 10 monclient(hunting): picked mon.noname-a con 0x7f9e5806e800 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:44.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e58133610 0x7f9e58117b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 10 monclient(hunting): picked mon.noname-b con 0x7f9e58133610 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:44.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5809ae80 con 0x7f9e5806e800 2024-09-15T03:58:44.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5809ea00 con 0x7f9e58133610 2024-09-15T03:58:44.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5811b260 con 0x7f9e5806e430 2024-09-15T03:58:44.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:44.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.186+0000 7f9e5f91d640 10 monclient(hunting): authenticate will time out at 747.830974s 2024-09-15T03:58:44.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5d6b9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e5806e430 0x7f9e5813f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e5806e800 0x7f9e5813f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e5806e800 0x7f9e5813f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56338/0 (socket says 172.21.15.110:56338) 2024-09-15T03:58:44.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5d6b9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e5806e430 0x7f9e5813f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54844/0 (socket says 172.21.15.110:54844) 2024-09-15T03:58:44.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 1 -- 172.21.15.110:0/4008831686 learned_addr learned my addr 172.21.15.110:0/4008831686 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:44.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 10 monclient(hunting): get_auth_request con 0x7f9e5806e800 auth_method 0 2024-09-15T03:58:44.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:44.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:44.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:44.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5d6b9640 10 monclient(hunting): get_auth_request con 0x7f9e5806e430 auth_method 0 2024-09-15T03:58:44.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5d6b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5d6b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5d6b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5d6b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5d6b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5d6b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 10 monclient(hunting): handle_auth_done global_id 4206 payload 824 2024-09-15T03:58:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 1 -- 172.21.15.110:0/4008831686 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e5806e430 msgr2=0x7f9e5813f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 1 --2- 172.21.15.110:0/4008831686 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e5806e430 0x7f9e5813f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 1 -- 172.21.15.110:0/4008831686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e58133610 msgr2=0x7f9e58117b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 1 --2- 172.21.15.110:0/4008831686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e58133610 0x7f9e58117b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 10 monclient: _finish_hunting 0 2024-09-15T03:58:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 1 monclient: found mon.noname-a 2024-09-15T03:58:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5ceb8640 1 -- 172.21.15.110:0/4008831686 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e50009a20 con 0x7f9e5806e800 2024-09-15T03:58:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.187+0000 7f9e5d6b9640 1 --2- 172.21.15.110:0/4008831686 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e5806e430 0x7f9e5813f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e5ceb8640 1 -- 172.21.15.110:0/4008831686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e5806e800 msgr2=0x7f9e5813f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e5ceb8640 1 --2- 172.21.15.110:0/4008831686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e5806e800 0x7f9e5813f920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009850 tx=0x7f9e48032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e4d7fa640 1 -- 172.21.15.110:0/4008831686 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48031a10 con 0x7f9e5806e800 2024-09-15T03:58:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e4d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e4d7fa640 10 monclient: dump: 2024-09-15T03:58:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:44.201 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:44.202 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e4d7fa640 10 monclient: _finish_auth 0 2024-09-15T03:58:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e4d7fa640 1 -- 172.21.15.110:0/4008831686 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e48031b70 con 0x7f9e5806e800 2024-09-15T03:58:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e4d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e4d7fa640 1 -- 172.21.15.110:0/4008831686 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4803d440 con 0x7f9e5806e800 2024-09-15T03:58:44.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e5f91d640 5 monclient: authenticate success, global_id 4206 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e5f91d640 10 monclient: _renew_subs 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e5f91d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e5f91d640 1 -- 172.21.15.110:0/4008831686 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e48009390 con 0x7f9e5806e800 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e4d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e4d7fa640 10 monclient: dump: 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:44.203 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:44.204 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e4f7fe640 4 set_mon_vals no callback set 2024-09-15T03:58:44.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e5f91d640 10 monclient: _renew_subs 2024-09-15T03:58:44.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e5f91d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:44.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.189+0000 7f9e5f91d640 1 -- 172.21.15.110:0/4008831686 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e58119c20 con 0x7f9e5806e800 2024-09-15T03:58:44.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.190+0000 7f9e4d7fa640 1 -- 172.21.15.110:0/4008831686 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f9e48039040 con 0x7f9e5806e800 2024-09-15T03:58:44.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.190+0000 7f9e4d7fa640 1 --2- 172.21.15.110:0/4008831686 >> v2:172.21.15.177:6812/28201 conn(0x7f9e3c038730 0x7f9e3c03abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.190+0000 7f9e5f91d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:44.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.190+0000 7f9e5f91d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:44.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.190+0000 7f9e5f91d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:44.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.190+0000 7f9e5f91d640 1 -- 172.21.15.110:0/4008831686 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e58070950 con 0x7f9e5806e800 2024-09-15T03:58:44.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.190+0000 7f9e4d7fa640 1 -- 172.21.15.110:0/4008831686 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f9e4806b0a0 con 0x7f9e5806e800 2024-09-15T03:58:44.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.190+0000 7f9e5d6b9640 1 --2- 172.21.15.110:0/4008831686 >> v2:172.21.15.177:6812/28201 conn(0x7f9e3c038730 0x7f9e3c03abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.190+0000 7f9e5d6b9640 10 monclient: get_auth_request con 0x7f9e3c038730 auth_method 0 2024-09-15T03:58:44.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.193+0000 7f9e5d6b9640 1 --2- 172.21.15.110:0/4008831686 >> v2:172.21.15.177:6812/28201 conn(0x7f9e3c038730 0x7f9e3c03abe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009e40 tx=0x7f9e54006f50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:44.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.195+0000 7f9e4d7fa640 1 -- 172.21.15.110:0/4008831686 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9e48042080 con 0x7f9e5806e800 2024-09-15T03:58:44.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.195+0000 7f9e4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:44.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.195+0000 7f9e4d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:44.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.389+0000 7f9e5f91d640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-15T03:58:44.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.389+0000 7f9e5f91d640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-15T03:58:44.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.389+0000 7f9e5f91d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:44.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.389+0000 7f9e5f91d640 1 -- 172.21.15.110:0/4008831686 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f9e5811b3a0 con 0x7f9e5806e800 2024-09-15T03:58:44.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.391+0000 7f9e4d7fa640 1 -- 172.21.15.110:0/4008831686 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+87608 (secure 0 0 0) 0x7f9e48037020 con 0x7f9e5806e800 2024-09-15T03:58:44.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.391+0000 7f9e4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-15T03:58:44.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.391+0000 7f9e4d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:44.394 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:44.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.393+0000 7f9e5f91d640 1 -- 172.21.15.110:0/4008831686 >> v2:172.21.15.177:6812/28201 conn(0x7f9e3c038730 msgr2=0x7f9e3c03abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.393+0000 7f9e5f91d640 1 --2- 172.21.15.110:0/4008831686 >> v2:172.21.15.177:6812/28201 conn(0x7f9e3c038730 0x7f9e3c03abe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009e40 tx=0x7f9e54006f50 comp rx=0 tx=0).stop 2024-09-15T03:58:44.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.393+0000 7f9e5f91d640 10 monclient: shutdown 2024-09-15T03:58:44.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.393+0000 7f9e5f91d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:44.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.393+0000 7f9e5f91d640 1 -- 172.21.15.110:0/4008831686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e5806e800 msgr2=0x7f9e5813f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.393+0000 7f9e5f91d640 1 --2- 172.21.15.110:0/4008831686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e5806e800 0x7f9e5813f920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009850 tx=0x7f9e48032000 comp rx=0 tx=0).stop 2024-09-15T03:58:44.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.393+0000 7f9e5f91d640 1 -- 172.21.15.110:0/4008831686 shutdown_connections 2024-09-15T03:58:44.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.393+0000 7f9e5f91d640 1 --2- 172.21.15.110:0/4008831686 >> v2:172.21.15.177:6812/28201 conn(0x7f9e3c038730 0x7f9e3c03abe0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.393+0000 7f9e5f91d640 1 --2- 172.21.15.110:0/4008831686 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e58133610 0x7f9e58117b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.393+0000 7f9e5f91d640 1 --2- 172.21.15.110:0/4008831686 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e5806e800 0x7f9e5813f920 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.393+0000 7f9e5f91d640 1 --2- 172.21.15.110:0/4008831686 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e5806e430 0x7f9e5813f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.393+0000 7f9e5f91d640 1 -- 172.21.15.110:0/4008831686 >> 172.21.15.110:0/4008831686 conn(0x7f9e5809d8f0 msgr2=0x7f9e5806d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:44.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.393+0000 7f9e5f91d640 1 -- 172.21.15.110:0/4008831686 shutdown_connections 2024-09-15T03:58:44.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.393+0000 7f9e5f91d640 1 -- 172.21.15.110:0/4008831686 wait complete. 2024-09-15T03:58:44.404 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":28201}]},"active_addr":"172.21.15.177:6812/28201","active_change":"2024-09-15T03:58:34.355076+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":379884208}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":2262029910}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":4149597931}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.177:0","nonce":3617759311}]}]} 2024-09-15T03:58:44.406 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-15T03:58:44.406 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T03:58:44.406 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:58:44.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.497+0000 7f851d64f640 10 monclient: get_monmap_and_config 2024-09-15T03:58:44.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.497+0000 7f851d64f640 10 monclient: build_initial_monmap 2024-09-15T03:58:44.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.497+0000 7f851d64f640 10 monclient: monmap: 2024-09-15T03:58:44.498 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:44.498 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.499 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:44.498650+0000 2024-09-15T03:58:44.499 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:44.498650+0000 2024-09-15T03:58:44.499 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:44.499 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.499 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:44.499 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:44.499 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:44.499 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:44.499 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:44.499 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.498+0000 7f851d64f640 1 Processor -- start 2024-09-15T03:58:44.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.498+0000 7f851d64f640 1 -- start start 2024-09-15T03:58:44.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.498+0000 7f851d64f640 10 monclient: init 2024-09-15T03:58:44.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.498+0000 7f851d64f640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:44.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.498+0000 7f851d64f640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-15T03:58:44.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.498+0000 7f851d64f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85180a1bf0 0x7f8518131810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.498+0000 7f851d64f640 10 monclient(hunting): picked mon.noname-b con 0x7f85180a1bf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:44.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.498+0000 7f851d64f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f851809e780 0x7f851809eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.498+0000 7f851d64f640 10 monclient(hunting): picked mon.noname-a con 0x7f851809e780 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:44.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.498+0000 7f851d64f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f851809f760 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:44.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f851d64f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851809c9b0 con 0x7f851809f760 2024-09-15T03:58:44.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f851d64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f851d64f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85180a0c20 con 0x7f851809e780 2024-09-15T03:58:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f851d64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f851d64f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851809cc90 con 0x7f85180a1bf0 2024-09-15T03:58:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f851d64f640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f851d64f640 10 monclient(hunting): authenticate will time out at 748.143030s 2024-09-15T03:58:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f851809e780 0x7f851809eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f85177fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f851809f760 0x7f851809fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33722/0 (socket says 172.21.15.110:33722) 2024-09-15T03:58:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f8516ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85180a1bf0 0x7f8518131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f85177fe640 1 -- 172.21.15.110:0/2104363560 learned_addr learned my addr 172.21.15.110:0/2104363560 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 10 monclient(hunting): get_auth_request con 0x7f851809e780 auth_method 0 2024-09-15T03:58:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f8516ffd640 10 monclient(hunting): get_auth_request con 0x7f85180a1bf0 auth_method 0 2024-09-15T03:58:44.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f8516ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:44.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f8516ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:44.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f8516ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:44.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:44.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:44.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:44.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f8516ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:44.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f8516ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:44.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f8516ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:44.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 10 monclient(hunting): handle_auth_done global_id 4209 payload 274 2024-09-15T03:58:44.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 1 -- 172.21.15.110:0/2104363560 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85180a1bf0 msgr2=0x7f8518131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:44.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 1 --2- 172.21.15.110:0/2104363560 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85180a1bf0 0x7f8518131810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 1 -- 172.21.15.110:0/2104363560 >> v1:172.21.15.110:6789/0 conn(0x7f851809f760 legacy=0x7f851809fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:44.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:44.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 1 monclient: found mon.noname-a 2024-09-15T03:58:44.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:44.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:44.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f850ffff640 1 -- 172.21.15.110:0/2104363560 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85180a1fc0 con 0x7f851809e780 2024-09-15T03:58:44.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.499+0000 7f8516ffd640 1 --2- 172.21.15.110:0/2104363560 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85180a1bf0 0x7f8518131810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:44.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f850ffff640 1 -- 172.21.15.110:0/2104363560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851809e780 msgr2=0x7f851809eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:44.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f850ffff640 1 --2- 172.21.15.110:0/2104363560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851809e780 0x7f851809eb50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8500009850 tx=0x7f85000090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22b0cfe881bc79b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:44.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f85167fc640 1 -- 172.21.15.110:0/2104363560 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850002ae80 con 0x7f851809e780 2024-09-15T03:58:44.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f85167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:44.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f85167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:44.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f85167fc640 10 monclient: dump: 2024-09-15T03:58:44.510 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:44.511 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.511 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.511 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.511 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:44.511 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.511 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:44.511 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:44.511 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:44.511 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f85167fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:44.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f85167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:44.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f85167fc640 1 -- 172.21.15.110:0/2104363560 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8500032440 con 0x7f851809e780 2024-09-15T03:58:44.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f851d64f640 5 monclient: authenticate success, global_id 4209 2024-09-15T03:58:44.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f851d64f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:44.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f85167fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:44.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f851d64f640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:44.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f85167fc640 1 -- 172.21.15.110:0/2104363560 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85000327c0 con 0x7f851809e780 2024-09-15T03:58:44.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f85167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:44.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f851d64f640 4 set_mon_vals no callback set 2024-09-15T03:58:44.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f85167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:44.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f85167fc640 10 monclient: dump: 2024-09-15T03:58:44.512 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:44.512 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.512 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.512 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.512 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:44.512 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.513 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:44.513 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:44.513 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:44.513 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f851d64f640 10 monclient: shutdown 2024-09-15T03:58:44.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f851d64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:44.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f851d64f640 1 -- 172.21.15.110:0/2104363560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851809e780 msgr2=0x7f851809eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f851d64f640 1 --2- 172.21.15.110:0/2104363560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851809e780 0x7f851809eb50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8500009850 tx=0x7f85000090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f851d64f640 1 -- 172.21.15.110:0/2104363560 shutdown_connections 2024-09-15T03:58:44.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f851d64f640 1 --2- 172.21.15.110:0/2104363560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851809e780 0x7f851809eb50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f851d64f640 1 --2- 172.21.15.110:0/2104363560 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85180a1bf0 0x7f8518131810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.502+0000 7f851d64f640 1 -- 172.21.15.110:0/2104363560 >> 172.21.15.110:0/2104363560 conn(0x7f8518062dd0 msgr2=0x7f85180631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:44.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.503+0000 7f851d64f640 1 -- 172.21.15.110:0/2104363560 shutdown_connections 2024-09-15T03:58:44.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.503+0000 7f851d64f640 1 -- 172.21.15.110:0/2104363560 wait complete. 2024-09-15T03:58:44.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.503+0000 7f851d64f640 10 monclient: build_initial_monmap 2024-09-15T03:58:44.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.503+0000 7f851d64f640 10 monclient: monmap: 2024-09-15T03:58:44.514 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:44.514 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:44.514 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:44.514 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:44.514 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:44.514 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.514 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:44.514 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:44.514 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:44.514 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.503+0000 7f851d64f640 1 Processor -- start 2024-09-15T03:58:44.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.503+0000 7f851d64f640 1 -- start start 2024-09-15T03:58:44.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.503+0000 7f851d64f640 10 monclient: init 2024-09-15T03:58:44.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:44.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:58:44.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851809e780 0x7f85180999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 10 monclient(hunting): picked mon.noname-a con 0x7f851809e780 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:44.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f851809f760 0x7f8518130670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 10 monclient(hunting): picked mon.noname-b con 0x7f851809f760 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:44.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85180a1bf0 0x7f851811b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 10 monclient(hunting): picked mon.noname-c con 0x7f85180a1bf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:44.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8518141d90 con 0x7f851809e780 2024-09-15T03:58:44.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851809ae00 con 0x7f851809f760 2024-09-15T03:58:44.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851811dd50 con 0x7f85180a1bf0 2024-09-15T03:58:44.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:44.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f851d64f640 10 monclient(hunting): authenticate will time out at 748.148230s 2024-09-15T03:58:44.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f8516ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851809e780 0x7f85180999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f8516ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851809e780 0x7f85180999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56358/0 (socket says 172.21.15.110:56358) 2024-09-15T03:58:44.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85180a1bf0 0x7f851811b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f8516ffd640 1 -- 172.21.15.110:0/554215052 learned_addr learned my addr 172.21.15.110:0/554215052 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:44.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f8516ffd640 10 monclient(hunting): get_auth_request con 0x7f851809e780 auth_method 0 2024-09-15T03:58:44.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f8516ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:44.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f8516ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:44.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f8516ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:44.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 10 monclient(hunting): get_auth_request con 0x7f85180a1bf0 auth_method 0 2024-09-15T03:58:44.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:44.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:44.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:44.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f8516ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:44.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f8516ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:44.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f8516ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:44.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:44.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:44.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:44.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 10 monclient(hunting): handle_auth_done global_id 4217 payload 824 2024-09-15T03:58:44.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f850ffff640 1 --2- 172.21.15.110:0/554215052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f851809f760 0x7f8518130670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 1 -- 172.21.15.110:0/554215052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f851809f760 msgr2=0x7f8518130670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 1 --2- 172.21.15.110:0/554215052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f851809f760 0x7f8518130670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 1 -- 172.21.15.110:0/554215052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851809e780 msgr2=0x7f85180999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 1 --2- 172.21.15.110:0/554215052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851809e780 0x7f85180999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 10 monclient: _finish_hunting 0 2024-09-15T03:58:44.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 1 monclient: found mon.noname-c 2024-09-15T03:58:44.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:44.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:44.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f85177fe640 1 -- 172.21.15.110:0/554215052 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8508009390 con 0x7f85180a1bf0 2024-09-15T03:58:44.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f850ffff640 10 monclient: get_auth_request con 0x7f851809f760 auth_method 0 2024-09-15T03:58:44.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f8516ffd640 1 --2- 172.21.15.110:0/554215052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851809e780 0x7f85180999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:44.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.504+0000 7f850ffff640 1 --2- 172.21.15.110:0/554215052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f851809f760 0x7f8518130670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:58:44.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f85177fe640 1 -- 172.21.15.110:0/554215052 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85180a1bf0 msgr2=0x7f851811b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:44.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f85177fe640 1 --2- 172.21.15.110:0/554215052 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85180a1bf0 0x7f851811b3d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f851000cb90 tx=0x7f85100052b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:44.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f850effd640 1 -- 172.21.15.110:0/554215052 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851000a850 con 0x7f85180a1bf0 2024-09-15T03:58:44.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f850effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f850effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f850effd640 10 monclient: dump: 2024-09-15T03:58:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:44.519 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f850effd640 10 monclient: _finish_auth 0 2024-09-15T03:58:44.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f850effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f850effd640 1 -- 172.21.15.110:0/554215052 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f851000a9d0 con 0x7f85180a1bf0 2024-09-15T03:58:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f851d64f640 5 monclient: authenticate success, global_id 4217 2024-09-15T03:58:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f851d64f640 10 monclient: _renew_subs 2024-09-15T03:58:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f851d64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f851d64f640 1 -- 172.21.15.110:0/554215052 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8500009390 con 0x7f85180a1bf0 2024-09-15T03:58:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f850effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f850effd640 1 -- 172.21.15.110:0/554215052 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851000ad90 con 0x7f85180a1bf0 2024-09-15T03:58:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f850effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f850effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f850effd640 10 monclient: dump: 2024-09-15T03:58:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.520 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:44.521 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.506+0000 7f8515ffb640 4 set_mon_vals no callback set 2024-09-15T03:58:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.507+0000 7f851d64f640 10 monclient: _renew_subs 2024-09-15T03:58:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.507+0000 7f851d64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.507+0000 7f851d64f640 1 -- 172.21.15.110:0/554215052 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f851811d790 con 0x7f85180a1bf0 2024-09-15T03:58:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.507+0000 7f850effd640 1 -- 172.21.15.110:0/554215052 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f8510007300 con 0x7f85180a1bf0 2024-09-15T03:58:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.507+0000 7f851d64f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:44.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.508+0000 7f850effd640 1 --2- 172.21.15.110:0/554215052 >> v2:172.21.15.177:6812/28201 conn(0x7f84f8038700 0x7f84f803abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.508+0000 7f851d64f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:44.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.508+0000 7f851d64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:44.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.508+0000 7f850effd640 1 -- 172.21.15.110:0/554215052 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f851004b660 con 0x7f85180a1bf0 2024-09-15T03:58:44.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.508+0000 7f851d64f640 1 -- 172.21.15.110:0/554215052 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f851811dec0 con 0x7f85180a1bf0 2024-09-15T03:58:44.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.508+0000 7f8516ffd640 1 --2- 172.21.15.110:0/554215052 >> v2:172.21.15.177:6812/28201 conn(0x7f84f8038700 0x7f84f803abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.508+0000 7f8516ffd640 10 monclient: get_auth_request con 0x7f84f8038700 auth_method 0 2024-09-15T03:58:44.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.510+0000 7f8516ffd640 1 --2- 172.21.15.110:0/554215052 >> v2:172.21.15.177:6812/28201 conn(0x7f84f8038700 0x7f84f803abb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8508011640 tx=0x7f85080090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:44.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.512+0000 7f850effd640 1 -- 172.21.15.110:0/554215052 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f851001b400 con 0x7f85180a1bf0 2024-09-15T03:58:44.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.512+0000 7f850effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:44.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.512+0000 7f850effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:44.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.666+0000 7f851d64f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:44.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.666+0000 7f851d64f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:44.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.666+0000 7f851d64f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:44.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.666+0000 7f851d64f640 1 -- 172.21.15.110:0/554215052 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8518070950 con 0x7f85180a1bf0 2024-09-15T03:58:44.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.666+0000 7f850effd640 1 -- 172.21.15.110:0/554215052 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+10108 (secure 0 0 0) 0x7f8510010af0 con 0x7f85180a1bf0 2024-09-15T03:58:44.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.666+0000 7f850effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:44.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.666+0000 7f850effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:44.668 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:44.668 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":19,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","created":"2024-09-15T03:58:28.265907+0000","modified":"2024-09-15T03:58:44.068637+0000","last_up_change":"2024-09-15T03:58:35.032799+0000","last_in_change":"2024-09-15T03:58:31.538597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:58:36.398831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:58:41.004777+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"105a3b3d-e410-47fb-ac54-6a5d5da4a46f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":28367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":28367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":28367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":28367}]},"public_addr":"172.21.15.110:6808/28367","cluster_addr":"172.21.15.110:6809/28367","heartbeat_back_addr":"172.21.15.110:6811/28367","heartbeat_front_addr":"172.21.15.110:6810/28367","state":["exists","up"]},{"osd":1,"uuid":"77f0f205-6c64-495a-8edd-043de180efa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":28369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":28369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":28369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":28369}]},"public_addr":"172.21.15.110:6800/28369","cluster_addr":"172.21.15.110:6801/28369","heartbeat_back_addr":"172.21.15.110:6803/28369","heartbeat_front_addr":"172.21.15.110:6802/28369","state":["exists","up"]},{"osd":2,"uuid":"e8a42031-303e-4ad0-9896-e5b84cc95cfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":28368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":28368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":28368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":28368}]},"public_addr":"172.21.15.110:6804/28368","cluster_addr":"172.21.15.110:6805/28368","heartbeat_back_addr":"172.21.15.110:6807/28368","heartbeat_front_addr":"172.21.15.110:6806/28368","state":["exists","up"]},{"osd":3,"uuid":"0d65db82-4a0e-4f31-a10a-dcd71dcb269f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":28628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6805","nonce":28628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6807","nonce":28628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":28628}]},"public_addr":"172.21.15.177:6804/28628","cluster_addr":"172.21.15.177:6805/28628","heartbeat_back_addr":"172.21.15.177:6807/28628","heartbeat_front_addr":"172.21.15.177:6806/28628","state":["exists","up"]},{"osd":4,"uuid":"e95e4f77-578d-4d03-bb4d-ee72b43bc227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":28629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6801","nonce":28629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6803","nonce":28629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":28629}]},"public_addr":"172.21.15.177:6800/28629","cluster_addr":"172.21.15.177:6801/28629","heartbeat_back_addr":"172.21.15.177:6803/28629","heartbeat_front_addr":"172.21.15.177:6802/28629","state":["exists","up"]},{"osd":5,"uuid":"f16ffee6-1e25-432c-b6c2-909a72ce24a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":28630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6809","nonce":28630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6811","nonce":28630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":28630}]},"public_addr":"172.21.15.177:6808/28630","cluster_addr":"172.21.15.177:6809/28630","heartbeat_back_addr":"172.21.15.177:6811/28630","heartbeat_front_addr":"172.21.15.177:6810/28630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.117560+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.137736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.130715+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.173663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.130009+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.172505+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:58:44.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.668+0000 7f851d64f640 1 -- 172.21.15.110:0/554215052 >> v2:172.21.15.177:6812/28201 conn(0x7f84f8038700 msgr2=0x7f84f803abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.668+0000 7f851d64f640 1 --2- 172.21.15.110:0/554215052 >> v2:172.21.15.177:6812/28201 conn(0x7f84f8038700 0x7f84f803abb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8508011640 tx=0x7f85080090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.668+0000 7f851d64f640 10 monclient: shutdown 2024-09-15T03:58:44.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.668+0000 7f851d64f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:44.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.668+0000 7f851d64f640 1 -- 172.21.15.110:0/554215052 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85180a1bf0 msgr2=0x7f851811b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.668+0000 7f851d64f640 1 --2- 172.21.15.110:0/554215052 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85180a1bf0 0x7f851811b3d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f851000cb90 tx=0x7f85100052b0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.668+0000 7f851d64f640 1 -- 172.21.15.110:0/554215052 shutdown_connections 2024-09-15T03:58:44.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.668+0000 7f851d64f640 1 --2- 172.21.15.110:0/554215052 >> v2:172.21.15.177:6812/28201 conn(0x7f84f8038700 0x7f84f803abb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.668+0000 7f851d64f640 1 --2- 172.21.15.110:0/554215052 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85180a1bf0 0x7f851811b3d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.668+0000 7f851d64f640 1 --2- 172.21.15.110:0/554215052 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f851809f760 0x7f8518130670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.668+0000 7f851d64f640 1 --2- 172.21.15.110:0/554215052 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f851809e780 0x7f85180999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.668+0000 7f851d64f640 1 -- 172.21.15.110:0/554215052 >> 172.21.15.110:0/554215052 conn(0x7f8518062dd0 msgr2=0x7f85180a3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:44.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.668+0000 7f851d64f640 1 -- 172.21.15.110:0/554215052 shutdown_connections 2024-09-15T03:58:44.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.668+0000 7f851d64f640 1 -- 172.21.15.110:0/554215052 wait complete. 2024-09-15T03:58:44.680 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T03:58:44.680 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:58:44.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.771+0000 7f879131d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:44.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.771+0000 7f879131d640 10 monclient: build_initial_monmap 2024-09-15T03:58:44.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.771+0000 7f879131d640 10 monclient: monmap: 2024-09-15T03:58:44.772 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:44.773 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.773 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:44.772691+0000 2024-09-15T03:58:44.773 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:44.772691+0000 2024-09-15T03:58:44.773 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:44.773 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.773 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:44.773 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:44.773 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:44.773 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:44.773 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:44.773 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 1 Processor -- start 2024-09-15T03:58:44.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 1 -- start start 2024-09-15T03:58:44.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 10 monclient: init 2024-09-15T03:58:44.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:44.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T03:58:44.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f878c0a1bf0 0x7f878c131810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 10 monclient(hunting): picked mon.noname-c con 0x7f878c0a1bf0 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:44.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f878c09e780 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:44.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f878c136a40 0x7f878c138e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 10 monclient(hunting): picked mon.noname-a con 0x7f878c136a40 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:44.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878c09c9b0 con 0x7f878c09e780 2024-09-15T03:58:44.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f878c0a0c20 con 0x7f878c136a40 2024-09-15T03:58:44.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f878c09cc90 con 0x7f878c0a1bf0 2024-09-15T03:58:44.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:44.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f879131d640 10 monclient(hunting): authenticate will time out at 748.416776s 2024-09-15T03:58:44.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f878b7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f878c136a40 0x7f878c138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f878a7fc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f878c09e780 0x7f878c09eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33728/0 (socket says 172.21.15.110:33728) 2024-09-15T03:58:44.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f878a7fc640 1 -- 172.21.15.110:0/2403834964 learned_addr learned my addr 172.21.15.110:0/2403834964 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:44.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f878b7fe640 10 monclient(hunting): get_auth_request con 0x7f878c136a40 auth_method 0 2024-09-15T03:58:44.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f878b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:44.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f878b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:44.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.772+0000 7f878b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:44.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878affd640 1 --2- 172.21.15.110:0/2403834964 >> v2:172.21.15.177:3300/0 conn(0x7f878c0a1bf0 0x7f878c131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:44.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:44.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:44.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878affd640 10 monclient(hunting): get_auth_request con 0x7f878c0a1bf0 auth_method 0 2024-09-15T03:58:44.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:44.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:44.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:44.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:44.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:44.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:44.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878b7fe640 10 monclient(hunting): handle_auth_done global_id 4215 payload 274 2024-09-15T03:58:44.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878b7fe640 1 -- 172.21.15.110:0/2403834964 >> v2:172.21.15.177:3300/0 conn(0x7f878c0a1bf0 msgr2=0x7f878c131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:44.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878b7fe640 1 --2- 172.21.15.110:0/2403834964 >> v2:172.21.15.177:3300/0 conn(0x7f878c0a1bf0 0x7f878c131810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878b7fe640 1 -- 172.21.15.110:0/2403834964 >> v1:172.21.15.110:6789/0 conn(0x7f878c09e780 legacy=0x7f878c09eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:44.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:58:44.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878b7fe640 1 monclient: found mon.noname-a 2024-09-15T03:58:44.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:44.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:44.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.773+0000 7f878b7fe640 1 -- 172.21.15.110:0/2403834964 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c0a1fc0 con 0x7f878c136a40 2024-09-15T03:58:44.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.775+0000 7f878b7fe640 1 -- 172.21.15.110:0/2403834964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f878c136a40 msgr2=0x7f878c138e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:44.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.775+0000 7f878b7fe640 1 --2- 172.21.15.110:0/2403834964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f878c136a40 0x7f878c138e20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f878c0a2bb0 tx=0x7f8784009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=859f49c5a2cdb832 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:44.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f8789ffb640 1 -- 172.21.15.110:0/2403834964 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8784007760 con 0x7f878c136a40 2024-09-15T03:58:44.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f8789ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:44.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f8789ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:44.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f8789ffb640 10 monclient: dump: 2024-09-15T03:58:44.785 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:44.785 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.785 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.785 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.785 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:44.785 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.785 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:44.786 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:44.786 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:44.786 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f8789ffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:44.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f8789ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:44.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f8789ffb640 1 -- 172.21.15.110:0/2403834964 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878402d050 con 0x7f878c136a40 2024-09-15T03:58:44.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f879131d640 5 monclient: authenticate success, global_id 4215 2024-09-15T03:58:44.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f879131d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:44.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f8789ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:44.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f879131d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:44.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f879131d640 4 set_mon_vals no callback set 2024-09-15T03:58:44.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f8789ffb640 1 -- 172.21.15.110:0/2403834964 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8784007b30 con 0x7f878c136a40 2024-09-15T03:58:44.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f8789ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:44.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f8789ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:44.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f8789ffb640 10 monclient: dump: 2024-09-15T03:58:44.787 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:44.787 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.787 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.787 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.787 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:44.787 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.787 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:44.787 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:44.787 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:44.787 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f879131d640 10 monclient: shutdown 2024-09-15T03:58:44.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f879131d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:44.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f879131d640 1 -- 172.21.15.110:0/2403834964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f878c136a40 msgr2=0x7f878c138e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f879131d640 1 --2- 172.21.15.110:0/2403834964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f878c136a40 0x7f878c138e20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f878c0a2bb0 tx=0x7f8784009780 comp rx=0 tx=0).stop 2024-09-15T03:58:44.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f879131d640 1 -- 172.21.15.110:0/2403834964 shutdown_connections 2024-09-15T03:58:44.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f879131d640 1 --2- 172.21.15.110:0/2403834964 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f878c136a40 0x7f878c138e20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f879131d640 1 --2- 172.21.15.110:0/2403834964 >> v2:172.21.15.177:3300/0 conn(0x7f878c0a1bf0 0x7f878c131810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f879131d640 1 -- 172.21.15.110:0/2403834964 >> 172.21.15.110:0/2403834964 conn(0x7f878c062dd0 msgr2=0x7f878c0631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:44.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f879131d640 1 -- 172.21.15.110:0/2403834964 shutdown_connections 2024-09-15T03:58:44.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.776+0000 7f879131d640 1 -- 172.21.15.110:0/2403834964 wait complete. 2024-09-15T03:58:44.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.777+0000 7f879131d640 10 monclient: build_initial_monmap 2024-09-15T03:58:44.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.777+0000 7f879131d640 10 monclient: monmap: 2024-09-15T03:58:44.788 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:44.788 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:44.789 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:44.789 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:44.789 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:44.789 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.789 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:44.789 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:44.789 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:44.789 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.777+0000 7f879131d640 1 Processor -- start 2024-09-15T03:58:44.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.777+0000 7f879131d640 1 -- start start 2024-09-15T03:58:44.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.777+0000 7f879131d640 10 monclient: init 2024-09-15T03:58:44.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:44.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:58:44.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f878c09e780 0x7f878c1410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 10 monclient(hunting): picked mon.noname-a con 0x7f878c09e780 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:44.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f878c0a1bf0 0x7f878c141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 10 monclient(hunting): picked mon.noname-b con 0x7f878c0a1bf0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:44.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f878c136a40 0x7f878c0999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 10 monclient(hunting): picked mon.noname-c con 0x7f878c136a40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:44.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c09c250 con 0x7f878c09e780 2024-09-15T03:58:44.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c0a2da0 con 0x7f878c0a1bf0 2024-09-15T03:58:44.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:44.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c11c700 con 0x7f878c136a40 2024-09-15T03:58:44.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:44.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f879131d640 10 monclient(hunting): authenticate will time out at 748.422119s 2024-09-15T03:58:44.790 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f878c09e780 0x7f878c1410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f878c09e780 0x7f878c1410e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56376/0 (socket says 172.21.15.110:56376) 2024-09-15T03:58:44.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 1 -- 172.21.15.110:0/1929367158 learned_addr learned my addr 172.21.15.110:0/1929367158 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:44.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 10 monclient(hunting): get_auth_request con 0x7f878c09e780 auth_method 0 2024-09-15T03:58:44.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:44.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:44.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:44.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878b7fe640 1 --2- 172.21.15.110:0/1929367158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f878c136a40 0x7f878c0999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878b7fe640 10 monclient(hunting): get_auth_request con 0x7f878c136a40 auth_method 0 2024-09-15T03:58:44.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:44.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:44.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:44.791 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:44.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:44.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:44.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:44.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:44.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:44.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 10 monclient(hunting): handle_auth_done global_id 4218 payload 824 2024-09-15T03:58:44.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878a7fc640 1 --2- 172.21.15.110:0/1929367158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f878c0a1bf0 0x7f878c141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 1 -- 172.21.15.110:0/1929367158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f878c136a40 msgr2=0x7f878c0999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 1 --2- 172.21.15.110:0/1929367158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f878c136a40 0x7f878c0999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 1 -- 172.21.15.110:0/1929367158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f878c0a1bf0 msgr2=0x7f878c141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 1 --2- 172.21.15.110:0/1929367158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f878c0a1bf0 0x7f878c141620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 10 monclient: _finish_hunting 0 2024-09-15T03:58:44.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 1 monclient: found mon.noname-a 2024-09-15T03:58:44.792 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:44.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:44.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.778+0000 7f878affd640 1 -- 172.21.15.110:0/1929367158 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8784009a20 con 0x7f878c09e780 2024-09-15T03:58:44.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f878affd640 1 -- 172.21.15.110:0/1929367158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f878c09e780 msgr2=0x7f878c1410e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:44.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f878affd640 1 --2- 172.21.15.110:0/1929367158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f878c09e780 0x7f878c1410e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f877c00ea20 tx=0x7f877c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:44.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f8772ffd640 1 -- 172.21.15.110:0/1929367158 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c00cc20 con 0x7f878c09e780 2024-09-15T03:58:44.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f8772ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:44.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f8772ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:44.793 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f8772ffd640 10 monclient: dump: 2024-09-15T03:58:44.793 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:44.793 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.793 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.793 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.793 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:44.794 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.794 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:44.794 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:44.794 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:44.794 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f8772ffd640 10 monclient: _finish_auth 0 2024-09-15T03:58:44.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f8772ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:44.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f879131d640 5 monclient: authenticate success, global_id 4218 2024-09-15T03:58:44.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f8772ffd640 1 -- 172.21.15.110:0/1929367158 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877c00cd80 con 0x7f878c09e780 2024-09-15T03:58:44.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f8772ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:44.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f8772ffd640 1 -- 172.21.15.110:0/1929367158 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c013680 con 0x7f878c09e780 2024-09-15T03:58:44.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f879131d640 10 monclient: _renew_subs 2024-09-15T03:58:44.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f879131d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:44.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f879131d640 1 -- 172.21.15.110:0/1929367158 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878c09b570 con 0x7f878c09e780 2024-09-15T03:58:44.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f8772ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:44.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f8772ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:44.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f8772ffd640 10 monclient: dump: 2024-09-15T03:58:44.795 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:44.795 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:44.795 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.795 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:44.795 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:44.795 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:44.795 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:44.795 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:44.795 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:44.795 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:44.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f87897fa640 4 set_mon_vals no callback set 2024-09-15T03:58:44.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f879131d640 10 monclient: _renew_subs 2024-09-15T03:58:44.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f879131d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:44.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.780+0000 7f879131d640 1 -- 172.21.15.110:0/1929367158 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f878c141b60 con 0x7f878c09e780 2024-09-15T03:58:44.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.781+0000 7f8772ffd640 1 -- 172.21.15.110:0/1929367158 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f877c013a70 con 0x7f878c09e780 2024-09-15T03:58:44.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.781+0000 7f879131d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:44.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.781+0000 7f879131d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:44.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.781+0000 7f879131d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:44.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.781+0000 7f879131d640 1 -- 172.21.15.110:0/1929367158 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f878c11cdd0 con 0x7f878c09e780 2024-09-15T03:58:44.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.781+0000 7f8772ffd640 1 --2- 172.21.15.110:0/1929367158 >> v2:172.21.15.177:6812/28201 conn(0x7f8778038730 0x7f877803abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:44.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.782+0000 7f8772ffd640 1 -- 172.21.15.110:0/1929367158 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f877c04b9b0 con 0x7f878c09e780 2024-09-15T03:58:44.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.782+0000 7f878a7fc640 1 --2- 172.21.15.110:0/1929367158 >> v2:172.21.15.177:6812/28201 conn(0x7f8778038730 0x7f877803abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:44.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.782+0000 7f878a7fc640 10 monclient: get_auth_request con 0x7f8778038730 auth_method 0 2024-09-15T03:58:44.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.784+0000 7f878a7fc640 1 --2- 172.21.15.110:0/1929367158 >> v2:172.21.15.177:6812/28201 conn(0x7f8778038730 0x7f877803abe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f878c0a11a0 tx=0x7f8780009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:44.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.786+0000 7f8772ffd640 1 -- 172.21.15.110:0/1929367158 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f877c0123d0 con 0x7f878c09e780 2024-09-15T03:58:44.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.786+0000 7f8772ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:44.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.786+0000 7f8772ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.940+0000 7f879131d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.940+0000 7f879131d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.940+0000 7f879131d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:44.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.940+0000 7f879131d640 1 -- 172.21.15.110:0/1929367158 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f878c070950 con 0x7f878c09e780 2024-09-15T03:58:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.941+0000 7f8772ffd640 1 -- 172.21.15.110:0/1929367158 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+10108 (secure 0 0 0) 0x7f877c011e20 con 0x7f878c09e780 2024-09-15T03:58:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.941+0000 7f8772ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:44.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.941+0000 7f8772ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:44.942 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:44.942 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":19,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","created":"2024-09-15T03:58:28.265907+0000","modified":"2024-09-15T03:58:44.068637+0000","last_up_change":"2024-09-15T03:58:35.032799+0000","last_in_change":"2024-09-15T03:58:31.538597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:58:36.398831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:58:41.004777+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"105a3b3d-e410-47fb-ac54-6a5d5da4a46f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":28367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":28367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":28367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":28367}]},"public_addr":"172.21.15.110:6808/28367","cluster_addr":"172.21.15.110:6809/28367","heartbeat_back_addr":"172.21.15.110:6811/28367","heartbeat_front_addr":"172.21.15.110:6810/28367","state":["exists","up"]},{"osd":1,"uuid":"77f0f205-6c64-495a-8edd-043de180efa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":28369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":28369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":28369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":28369}]},"public_addr":"172.21.15.110:6800/28369","cluster_addr":"172.21.15.110:6801/28369","heartbeat_back_addr":"172.21.15.110:6803/28369","heartbeat_front_addr":"172.21.15.110:6802/28369","state":["exists","up"]},{"osd":2,"uuid":"e8a42031-303e-4ad0-9896-e5b84cc95cfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":28368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":28368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":28368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":28368}]},"public_addr":"172.21.15.110:6804/28368","cluster_addr":"172.21.15.110:6805/28368","heartbeat_back_addr":"172.21.15.110:6807/28368","heartbeat_front_addr":"172.21.15.110:6806/28368","state":["exists","up"]},{"osd":3,"uuid":"0d65db82-4a0e-4f31-a10a-dcd71dcb269f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":28628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6805","nonce":28628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6807","nonce":28628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":28628}]},"public_addr":"172.21.15.177:6804/28628","cluster_addr":"172.21.15.177:6805/28628","heartbeat_back_addr":"172.21.15.177:6807/28628","heartbeat_front_addr":"172.21.15.177:6806/28628","state":["exists","up"]},{"osd":4,"uuid":"e95e4f77-578d-4d03-bb4d-ee72b43bc227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":28629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6801","nonce":28629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6803","nonce":28629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":28629}]},"public_addr":"172.21.15.177:6800/28629","cluster_addr":"172.21.15.177:6801/28629","heartbeat_back_addr":"172.21.15.177:6803/28629","heartbeat_front_addr":"172.21.15.177:6802/28629","state":["exists","up"]},{"osd":5,"uuid":"f16ffee6-1e25-432c-b6c2-909a72ce24a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":28630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6809","nonce":28630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6811","nonce":28630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":28630}]},"public_addr":"172.21.15.177:6808/28630","cluster_addr":"172.21.15.177:6809/28630","heartbeat_back_addr":"172.21.15.177:6811/28630","heartbeat_front_addr":"172.21.15.177:6810/28630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.117560+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.137736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.130715+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.173663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.130009+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.172505+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:58:44.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.943+0000 7f879131d640 1 -- 172.21.15.110:0/1929367158 >> v2:172.21.15.177:6812/28201 conn(0x7f8778038730 msgr2=0x7f877803abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.943+0000 7f879131d640 1 --2- 172.21.15.110:0/1929367158 >> v2:172.21.15.177:6812/28201 conn(0x7f8778038730 0x7f877803abe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f878c0a11a0 tx=0x7f8780009040 comp rx=0 tx=0).stop 2024-09-15T03:58:44.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.943+0000 7f879131d640 10 monclient: shutdown 2024-09-15T03:58:44.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.943+0000 7f879131d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:44.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.943+0000 7f879131d640 1 -- 172.21.15.110:0/1929367158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f878c09e780 msgr2=0x7f878c1410e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:44.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.943+0000 7f879131d640 1 --2- 172.21.15.110:0/1929367158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f878c09e780 0x7f878c1410e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f877c00ea20 tx=0x7f877c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.943+0000 7f879131d640 1 -- 172.21.15.110:0/1929367158 shutdown_connections 2024-09-15T03:58:44.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.943+0000 7f879131d640 1 --2- 172.21.15.110:0/1929367158 >> v2:172.21.15.177:6812/28201 conn(0x7f8778038730 0x7f877803abe0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.943+0000 7f879131d640 1 --2- 172.21.15.110:0/1929367158 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f878c136a40 0x7f878c0999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.943+0000 7f879131d640 1 --2- 172.21.15.110:0/1929367158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f878c0a1bf0 0x7f878c141620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.943+0000 7f879131d640 1 --2- 172.21.15.110:0/1929367158 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f878c09e780 0x7f878c1410e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:44.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.943+0000 7f879131d640 1 -- 172.21.15.110:0/1929367158 >> 172.21.15.110:0/1929367158 conn(0x7f878c062dd0 msgr2=0x7f878c0a4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:44.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.943+0000 7f879131d640 1 -- 172.21.15.110:0/1929367158 shutdown_connections 2024-09-15T03:58:44.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:44.943+0000 7f879131d640 1 -- 172.21.15.110:0/1929367158 wait complete. 2024-09-15T03:58:44.954 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T03:58:44.955 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T03:58:44.955 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T03:58:44.956 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T03:58:44.956 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-15T03:58:44.956 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-15T03:58:45.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.062+0000 7f9af9709640 10 monclient: get_monmap_and_config 2024-09-15T03:58:45.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.062+0000 7f9af9709640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.062+0000 7f9af9709640 10 monclient: monmap: 2024-09-15T03:58:45.064 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.064 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.064 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:45.063208+0000 2024-09-15T03:58:45.064 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:45.063208+0000 2024-09-15T03:58:45.064 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.064 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.064 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:45.064 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:45.064 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:45.064 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:45.064 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:45.064 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.062+0000 7f9af9709640 1 Processor -- start 2024-09-15T03:58:45.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.062+0000 7f9af9709640 1 -- start start 2024-09-15T03:58:45.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.062+0000 7f9af9709640 10 monclient: init 2024-09-15T03:58:45.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7fbc52d1d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:45.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7f9af9709640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7f9af9709640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-15T03:58:45.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7fbc52d1d640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7fbc52d1d640 10 monclient: monmap: 2024-09-15T03:58:45.066 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.066 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.066 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:45.064397+0000 2024-09-15T03:58:45.066 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:45.064397+0000 2024-09-15T03:58:45.066 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.066 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.066 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:45.066 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:45.066 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:45.067 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:45.067 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:45.067 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7f9af9709640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9af40a1a10 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:45.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7f9af9709640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9af409ee80 0x7f9af4131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7f9af9709640 10 monclient(hunting): picked mon.noname-a con 0x7f9af409ee80 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:45.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7f9af9709640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9af41369e0 0x7f9af4138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7f9af9709640 10 monclient(hunting): picked mon.noname-b con 0x7f9af41369e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7f9af9709640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af409f2c0 con 0x7f9af40a1a10 2024-09-15T03:58:45.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7f9af9709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7f9af9709640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9af40a0850 con 0x7f9af409ee80 2024-09-15T03:58:45.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7f9af9709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7f9af9709640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af409c9a0 con 0x7f9af41369e0 2024-09-15T03:58:45.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af9709640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af9709640 10 monclient(hunting): authenticate will time out at 748.708025s 2024-09-15T03:58:45.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.063+0000 7f9af2ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9af40a1a10 0x7f9af409e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33734/0 (socket says 172.21.15.110:33734) 2024-09-15T03:58:45.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af2ffd640 1 -- 172.21.15.110:0/1099087805 learned_addr learned my addr 172.21.15.110:0/1099087805 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:45.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 1 --2- 172.21.15.110:0/1099087805 >> v2:172.21.15.110:3300/0 conn(0x7f9af409ee80 0x7f9af4131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 10 monclient(hunting): get_auth_request con 0x7f9af409ee80 auth_method 0 2024-09-15T03:58:45.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 10 monclient(hunting): handle_auth_done global_id 4221 payload 274 2024-09-15T03:58:45.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 1 -- 172.21.15.110:0/1099087805 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9af41369e0 msgr2=0x7f9af4138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7fbc52d1d640 1 Processor -- start 2024-09-15T03:58:45.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af37fe640 1 --2- 172.21.15.110:0/1099087805 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9af41369e0 0x7f9af4138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 1 --2- 172.21.15.110:0/1099087805 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9af41369e0 0x7f9af4138dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 1 -- 172.21.15.110:0/1099087805 >> v1:172.21.15.110:6789/0 conn(0x7f9af40a1a10 legacy=0x7f9af409e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 1 monclient: found mon.noname-a 2024-09-15T03:58:45.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7fbc52d1d640 1 -- start start 2024-09-15T03:58:45.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7fbc52d1d640 10 monclient: init 2024-09-15T03:58:45.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af27fc640 1 -- 172.21.15.110:0/1099087805 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af409cc80 con 0x7f9af409ee80 2024-09-15T03:58:45.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.065+0000 7fbc52d1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.065+0000 7fbc52d1d640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T03:58:45.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af37fe640 10 monclient: get_auth_request con 0x7f9af41369e0 auth_method 0 2024-09-15T03:58:45.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.064+0000 7f9af37fe640 1 --2- 172.21.15.110:0/1099087805 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9af41369e0 0x7f9af4138dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:58:45.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.065+0000 7fbc52d1d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fbc4c0a1a10 0x7fbc4c131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.065+0000 7fbc52d1d640 10 monclient(hunting): picked mon.noname-c con 0x7fbc4c0a1a10 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:45.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.065+0000 7fbc52d1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09e770 0x7fbc4c09eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.065+0000 7fbc52d1d640 10 monclient(hunting): picked mon.noname-b con 0x7fbc4c09e770 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.065+0000 7fbc52d1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc4c09f800 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:45.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.065+0000 7fbc52d1d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c0a0850 con 0x7fbc4c09f800 2024-09-15T03:58:45.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.065+0000 7fbc4bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09e770 0x7fbc4c09eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc52d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc52d1d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4c09c9a0 con 0x7fbc4c0a1a10 2024-09-15T03:58:45.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc52d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc52d1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c09cc80 con 0x7fbc4c09e770 2024-09-15T03:58:45.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc52d1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc52d1d640 10 monclient(hunting): authenticate will time out at 748.710036s 2024-09-15T03:58:45.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09e770 0x7fbc4c09eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54898/0 (socket says 172.21.15.110:54898) 2024-09-15T03:58:45.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 1 -- 172.21.15.110:0/1387714487 learned_addr learned my addr 172.21.15.110:0/1387714487 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 10 monclient(hunting): get_auth_request con 0x7fbc4c09e770 auth_method 0 2024-09-15T03:58:45.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc50ab9640 1 --2- 172.21.15.110:0/1387714487 >> v2:172.21.15.177:3300/0 conn(0x7fbc4c0a1a10 0x7fbc4c131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc50ab9640 10 monclient(hunting): get_auth_request con 0x7fbc4c0a1a10 auth_method 0 2024-09-15T03:58:45.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc50ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc50ab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc50ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc50ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc50ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc50ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2024-09-15T03:58:45.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 1 -- 172.21.15.110:0/1387714487 >> v2:172.21.15.177:3300/0 conn(0x7fbc4c0a1a10 msgr2=0x7fbc4c131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 1 --2- 172.21.15.110:0/1387714487 >> v2:172.21.15.177:3300/0 conn(0x7fbc4c0a1a10 0x7fbc4c131b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 1 -- 172.21.15.110:0/1387714487 >> v1:172.21.15.110:6789/0 conn(0x7fbc4c09f800 legacy=0x7fbc4c09fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 1 monclient: found mon.noname-b 2024-09-15T03:58:45.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.066+0000 7fbc4bfff640 1 -- 172.21.15.110:0/1387714487 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c09c3f0 con 0x7fbc4c09e770 2024-09-15T03:58:45.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.067+0000 7f9af27fc640 1 -- 172.21.15.110:0/1099087805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9af409ee80 msgr2=0x7f9af4131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.067+0000 7f9af27fc640 1 --2- 172.21.15.110:0/1099087805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9af409ee80 0x7f9af4131b90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8009850 tx=0x7f9ae80090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5a076257d0fe8a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af1ffb640 1 -- 172.21.15.110:0/1099087805 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae8032440 con 0x7f9af409ee80 2024-09-15T03:58:45.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:45.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af1ffb640 10 monclient: dump: 2024-09-15T03:58:45.080 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.081 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.081 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.081 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.081 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.081 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.081 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.081 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.081 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.081 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af1ffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af1ffb640 1 -- 172.21.15.110:0/1099087805 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae80325a0 con 0x7f9af409ee80 2024-09-15T03:58:45.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af1ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af1ffb640 1 -- 172.21.15.110:0/1099087805 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae8032940 con 0x7f9af409ee80 2024-09-15T03:58:45.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af9709640 5 monclient: authenticate success, global_id 4221 2024-09-15T03:58:45.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af9709640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:45.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af9709640 4 set_mon_vals no callback set 2024-09-15T03:58:45.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:45.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af1ffb640 10 monclient: dump: 2024-09-15T03:58:45.082 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.082 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.082 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.082 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.082 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.082 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.082 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.082 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.083 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.083 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af9709640 10 monclient: shutdown 2024-09-15T03:58:45.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af9709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af9709640 1 -- 172.21.15.110:0/1099087805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9af409ee80 msgr2=0x7f9af4131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.068+0000 7f9af9709640 1 --2- 172.21.15.110:0/1099087805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9af409ee80 0x7f9af4131b90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8009850 tx=0x7f9ae80090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7f9af9709640 1 -- 172.21.15.110:0/1099087805 shutdown_connections 2024-09-15T03:58:45.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7f9af9709640 1 --2- 172.21.15.110:0/1099087805 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9af41369e0 0x7f9af4138dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7f9af9709640 1 --2- 172.21.15.110:0/1099087805 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9af409ee80 0x7f9af4131b90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7f9af9709640 1 -- 172.21.15.110:0/1099087805 >> 172.21.15.110:0/1099087805 conn(0x7f9af406e860 msgr2=0x7f9af406ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc4bfff640 1 -- 172.21.15.110:0/1387714487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09e770 msgr2=0x7fbc4c09eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc4bfff640 1 --2- 172.21.15.110:0/1387714487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09e770 0x7fbc4c09eb80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbc44009f20 tx=0x7fbc44009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=247a729bb40b5c2a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7f9af9709640 1 -- 172.21.15.110:0/1099087805 shutdown_connections 2024-09-15T03:58:45.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7f9af9709640 1 -- 172.21.15.110:0/1099087805 wait complete. 2024-09-15T03:58:45.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc4b7fe640 1 -- 172.21.15.110:0/1387714487 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc44007610 con 0x7fbc4c09e770 2024-09-15T03:58:45.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc4b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:45.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc4b7fe640 10 monclient: dump: 2024-09-15T03:58:45.085 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.085 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.085 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.085 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.085 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.085 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.085 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.085 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.085 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.085 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc4b7fe640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc4b7fe640 1 -- 172.21.15.110:0/1387714487 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4402d050 con 0x7fbc4c09e770 2024-09-15T03:58:45.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc4b7fe640 1 -- 172.21.15.110:0/1387714487 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc440079a0 con 0x7fbc4c09e770 2024-09-15T03:58:45.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc4b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:45.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc4b7fe640 10 monclient: dump: 2024-09-15T03:58:45.086 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.086 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.086 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.086 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.086 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.086 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.086 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.086 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.086 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.087 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc52d1d640 5 monclient: authenticate success, global_id 4223 2024-09-15T03:58:45.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc52d1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:45.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.069+0000 7fbc52d1d640 4 set_mon_vals no callback set 2024-09-15T03:58:45.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7f9af9709640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7fbc52d1d640 10 monclient: shutdown 2024-09-15T03:58:45.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7fbc52d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7fbc52d1d640 1 -- 172.21.15.110:0/1387714487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09e770 msgr2=0x7fbc4c09eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7fbc52d1d640 1 --2- 172.21.15.110:0/1387714487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09e770 0x7fbc4c09eb80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbc44009f20 tx=0x7fbc44009840 comp rx=0 tx=0).stop 2024-09-15T03:58:45.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7f9af9709640 10 monclient: monmap: 2024-09-15T03:58:45.088 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.088 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:45.088 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:45.088 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:45.088 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.088 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.088 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:45.088 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:45.088 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:45.089 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7fbc52d1d640 1 -- 172.21.15.110:0/1387714487 shutdown_connections 2024-09-15T03:58:45.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7fbc52d1d640 1 --2- 172.21.15.110:0/1387714487 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09e770 0x7fbc4c09eb80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7fbc52d1d640 1 --2- 172.21.15.110:0/1387714487 >> v2:172.21.15.177:3300/0 conn(0x7fbc4c0a1a10 0x7fbc4c131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7fbc52d1d640 1 -- 172.21.15.110:0/1387714487 >> 172.21.15.110:0/1387714487 conn(0x7fbc4c06e860 msgr2=0x7fbc4c06ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7fbc52d1d640 1 -- 172.21.15.110:0/1387714487 shutdown_connections 2024-09-15T03:58:45.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7fbc52d1d640 1 -- 172.21.15.110:0/1387714487 wait complete. 2024-09-15T03:58:45.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7f9af9709640 1 Processor -- start 2024-09-15T03:58:45.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7f9af9709640 1 -- start start 2024-09-15T03:58:45.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7f9af9709640 10 monclient: init 2024-09-15T03:58:45.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7fbc52d1d640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.070+0000 7fbc52d1d640 10 monclient: monmap: 2024-09-15T03:58:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:45.091 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7fbc52d1d640 1 Processor -- start 2024-09-15T03:58:45.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:58:45.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9af409ee80 0x7f9af4097480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 10 monclient(hunting): picked mon.noname-a con 0x7f9af409ee80 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:45.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9af40a1a10 0x7f9af40979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 10 monclient(hunting): picked mon.noname-b con 0x7f9af40a1a10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:45.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af2ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9af409ee80 0x7f9af4097480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7fbc52d1d640 1 -- start start 2024-09-15T03:58:45.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7fbc52d1d640 10 monclient: init 2024-09-15T03:58:45.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af2ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9af409ee80 0x7f9af4097480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56400/0 (socket says 172.21.15.110:56400) 2024-09-15T03:58:45.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af2ffd640 1 -- 172.21.15.110:0/3663603483 learned_addr learned my addr 172.21.15.110:0/3663603483 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 1 --2- 172.21.15.110:0/3663603483 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9af41369e0 0x7f9af4097f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7fbc52d1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7fbc52d1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:58:45.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7fbc52d1d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc4c09e770 0x7fbc4c09a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7fbc52d1d640 10 monclient(hunting): picked mon.noname-a con 0x7fbc4c09e770 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:45.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7fbc52d1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09f800 0x7fbc4c06bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc52d1d640 10 monclient(hunting): picked mon.noname-c con 0x7fbc4c09f800 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc52d1d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc4c0a1a10 0x7fbc4c06a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc52d1d640 10 monclient(hunting): picked mon.noname-b con 0x7fbc4c0a1a10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:45.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc52d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc52d1d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c1348b0 con 0x7fbc4c09e770 2024-09-15T03:58:45.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc52d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc52d1d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c06cdc0 con 0x7fbc4c0a1a10 2024-09-15T03:58:45.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc52d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc52d1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4c06c4d0 con 0x7fbc4c09f800 2024-09-15T03:58:45.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc52d1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc52d1d640 10 monclient(hunting): authenticate will time out at 748.716115s 2024-09-15T03:58:45.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09f800 0x7fbc4c06bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09f800 0x7fbc4c06bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54916/0 (socket says 172.21.15.110:54916) 2024-09-15T03:58:45.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 1 -- 172.21.15.110:0/2637460483 learned_addr learned my addr 172.21.15.110:0/2637460483 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 10 monclient(hunting): get_auth_request con 0x7fbc4c09f800 auth_method 0 2024-09-15T03:58:45.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc50ab9640 1 --2- 172.21.15.110:0/2637460483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc4c09e770 0x7fbc4c09a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc50ab9640 10 monclient(hunting): get_auth_request con 0x7fbc4c09e770 auth_method 0 2024-09-15T03:58:45.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc50ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc50ab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc50ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc50ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc50ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc50ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 10 monclient(hunting): picked mon.noname-c con 0x7f9af41369e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc512ba640 1 --2- 172.21.15.110:0/2637460483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc4c0a1a10 0x7fbc4c06a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc512ba640 10 monclient(hunting): get_auth_request con 0x7fbc4c0a1a10 auth_method 0 2024-09-15T03:58:45.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc512ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc512ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc512ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 1 -- 172.21.15.110:0/3663603483 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af40a11c0 con 0x7f9af409ee80 2024-09-15T03:58:45.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 1 -- 172.21.15.110:0/3663603483 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af4138f00 con 0x7f9af40a1a10 2024-09-15T03:58:45.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2024-09-15T03:58:45.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 1 -- 172.21.15.110:0/2637460483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc4c0a1a10 msgr2=0x7fbc4c06a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 1 --2- 172.21.15.110:0/2637460483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbc4c0a1a10 0x7fbc4c06a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 1 -- 172.21.15.110:0/2637460483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc4c09e770 msgr2=0x7fbc4c09a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 1 --2- 172.21.15.110:0/2637460483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbc4c09e770 0x7fbc4c09a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 1 monclient: found mon.noname-c 2024-09-15T03:58:45.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7fbc4bfff640 1 -- 172.21.15.110:0/2637460483 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc44009ae0 con 0x7fbc4c09f800 2024-09-15T03:58:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 1 -- 172.21.15.110:0/3663603483 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af409a890 con 0x7f9af41369e0 2024-09-15T03:58:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af9709640 10 monclient(hunting): authenticate will time out at 748.715610s 2024-09-15T03:58:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af2ffd640 10 monclient(hunting): get_auth_request con 0x7f9af409ee80 auth_method 0 2024-09-15T03:58:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af37fe640 1 --2- 172.21.15.110:0/3663603483 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9af41369e0 0x7f9af4097f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af37fe640 10 monclient(hunting): get_auth_request con 0x7f9af41369e0 auth_method 0 2024-09-15T03:58:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af27fc640 1 --2- 172.21.15.110:0/3663603483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9af40a1a10 0x7f9af40979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.071+0000 7f9af37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7f9af37fe640 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2024-09-15T03:58:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7f9af37fe640 1 -- 172.21.15.110:0/3663603483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9af40a1a10 msgr2=0x7f9af40979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7f9af37fe640 1 --2- 172.21.15.110:0/3663603483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9af40a1a10 0x7f9af40979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7f9af37fe640 1 -- 172.21.15.110:0/3663603483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9af409ee80 msgr2=0x7f9af4097480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7f9af37fe640 1 --2- 172.21.15.110:0/3663603483 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9af409ee80 0x7f9af4097480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7f9af37fe640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7f9af37fe640 1 monclient: found mon.noname-c 2024-09-15T03:58:45.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7f9af37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7f9af37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7f9af37fe640 1 -- 172.21.15.110:0/3663603483 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae8009390 con 0x7f9af41369e0 2024-09-15T03:58:45.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7f9af27fc640 10 monclient: get_auth_request con 0x7f9af40a1a10 auth_method 0 2024-09-15T03:58:45.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.072+0000 7f9af27fc640 1 --2- 172.21.15.110:0/3663603483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9af40a1a10 0x7f9af40979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:58:45.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.074+0000 7f9af37fe640 1 -- 172.21.15.110:0/3663603483 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9af41369e0 msgr2=0x7f9af4097f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.074+0000 7f9af37fe640 1 --2- 172.21.15.110:0/3663603483 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9af41369e0 0x7f9af4097f00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00cf30 tx=0x7f9aec007840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9adaffd640 1 -- 172.21.15.110:0/3663603483 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec00ae30 con 0x7f9af41369e0 2024-09-15T03:58:45.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc4bfff640 1 -- 172.21.15.110:0/2637460483 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09f800 msgr2=0x7fbc4c06bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9adaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9adaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:45.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9adaffd640 10 monclient: dump: 2024-09-15T03:58:45.105 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.105 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.105 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.106 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9adaffd640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9adaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9adaffd640 1 -- 172.21.15.110:0/3663603483 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec008750 con 0x7f9af41369e0 2024-09-15T03:58:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9adaffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9adaffd640 1 -- 172.21.15.110:0/3663603483 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec008b10 con 0x7f9af41369e0 2024-09-15T03:58:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9adaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9adaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:45.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9adaffd640 10 monclient: dump: 2024-09-15T03:58:45.107 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.107 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.107 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.107 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.107 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.107 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.107 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.107 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.107 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.107 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9af9709640 5 monclient: authenticate success, global_id 4226 2024-09-15T03:58:45.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc4bfff640 1 --2- 172.21.15.110:0/2637460483 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09f800 0x7fbc4c06bf90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbc4402ac70 tx=0x7fbc44036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9af9709640 10 monclient: _renew_subs 2024-09-15T03:58:45.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9af9709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9af9709640 1 -- 172.21.15.110:0/3663603483 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af409b100 con 0x7f9af41369e0 2024-09-15T03:58:45.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9af17fa640 4 set_mon_vals no callback set 2024-09-15T03:58:45.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc48ff9640 1 -- 172.21.15.110:0/2637460483 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4403b900 con 0x7fbc4c09f800 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc48ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc48ff9640 10 monclient: dump: 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc48ff9640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc52d1d640 5 monclient: authenticate success, global_id 4229 2024-09-15T03:58:45.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc52d1d640 10 monclient: _renew_subs 2024-09-15T03:58:45.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc52d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc52d1d640 1 -- 172.21.15.110:0/2637460483 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4c09b2f0 con 0x7fbc4c09f800 2024-09-15T03:58:45.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9af9709640 10 monclient: _renew_subs 2024-09-15T03:58:45.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc52d1d640 10 monclient: _renew_subs 2024-09-15T03:58:45.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc52d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7fbc52d1d640 1 -- 172.21.15.110:0/2637460483 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4c06da90 con 0x7fbc4c09f800 2024-09-15T03:58:45.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9af9709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.075+0000 7f9af9709640 1 -- 172.21.15.110:0/3663603483 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af4099ae0 con 0x7f9af41369e0 2024-09-15T03:58:45.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.076+0000 7fbc48ff9640 1 -- 172.21.15.110:0/2637460483 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4403ba60 con 0x7fbc4c09f800 2024-09-15T03:58:45.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.076+0000 7fbc48ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.076+0000 7fbc48ff9640 1 -- 172.21.15.110:0/2637460483 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4403be20 con 0x7fbc4c09f800 2024-09-15T03:58:45.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.076+0000 7fbc48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.076+0000 7fbc48ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:45.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.076+0000 7fbc48ff9640 10 monclient: dump: 2024-09-15T03:58:45.114 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.114 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.114 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.114 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.114 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.114 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.114 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.114 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.114 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.115 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.076+0000 7fbc4affd640 4 set_mon_vals no callback set 2024-09-15T03:58:45.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.077+0000 7fbc52d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.077+0000 7fbc52d1d640 1 -- 172.21.15.110:0/2637460483 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbc2c000f20 con 0x7fbc4c09f800 2024-09-15T03:58:45.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.077+0000 7f9adaffd640 1 -- 172.21.15.110:0/3663603483 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f9aec0263b0 con 0x7f9af41369e0 2024-09-15T03:58:45.115 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.077+0000 7f9af9709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.077+0000 7f9af9709640 1 -- 172.21.15.110:0/3663603483 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9af40647e0 con 0x7f9af41369e0 2024-09-15T03:58:45.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.077+0000 7fbc48ff9640 1 -- 172.21.15.110:0/2637460483 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fbc44038070 con 0x7fbc4c09f800 2024-09-15T03:58:45.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.078+0000 7fbc48ff9640 1 --2- 172.21.15.110:0/2637460483 >> v2:172.21.15.177:6812/28201 conn(0x7fbc38038700 0x7fbc3803abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.078+0000 7fbc48ff9640 1 -- 172.21.15.110:0/2637460483 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7fbc4406bd40 con 0x7fbc4c09f800 2024-09-15T03:58:45.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.078+0000 7fbc48ff9640 1 --2- 172.21.15.110:0/2637460483 >> v2:172.21.15.177:6808/28630 conn(0x7fbc3803e1a0 0x7fbc380405d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.078+0000 7fbc48ff9640 1 -- 172.21.15.110:0/2637460483 --> v2:172.21.15.177:6808/28630 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbc38040c60 con 0x7fbc3803e1a0 2024-09-15T03:58:45.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.078+0000 7fbc512ba640 1 --2- 172.21.15.110:0/2637460483 >> v2:172.21.15.177:6808/28630 conn(0x7fbc3803e1a0 0x7fbc380405d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.078+0000 7fbc512ba640 10 monclient: get_auth_request con 0x7fbc3803e1a0 auth_method 0 2024-09-15T03:58:45.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.078+0000 7f9adaffd640 1 --2- 172.21.15.110:0/3663603483 >> v2:172.21.15.177:6812/28201 conn(0x7f9ae0038750 0x7f9ae003ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.078+0000 7f9adaffd640 1 -- 172.21.15.110:0/3663603483 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f9aec054b40 con 0x7f9af41369e0 2024-09-15T03:58:45.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.078+0000 7f9adaffd640 1 --2- 172.21.15.110:0/3663603483 >> v2:172.21.15.177:6804/28628 conn(0x7f9ae003e270 0x7f9ae00406a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.078+0000 7f9adaffd640 1 -- 172.21.15.110:0/3663603483 --> v2:172.21.15.177:6804/28628 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ae0040d30 con 0x7f9ae003e270 2024-09-15T03:58:45.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.079+0000 7fbc50ab9640 1 --2- 172.21.15.110:0/2637460483 >> v2:172.21.15.177:6812/28201 conn(0x7fbc38038700 0x7fbc3803abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.079+0000 7fbc50ab9640 10 monclient: get_auth_request con 0x7fbc38038700 auth_method 0 2024-09-15T03:58:45.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.079+0000 7f9af2ffd640 1 --2- 172.21.15.110:0/3663603483 >> v2:172.21.15.177:6812/28201 conn(0x7f9ae0038750 0x7f9ae003ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.079+0000 7f9af2ffd640 10 monclient: get_auth_request con 0x7f9ae0038750 auth_method 0 2024-09-15T03:58:45.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.079+0000 7f9af27fc640 1 --2- 172.21.15.110:0/3663603483 >> v2:172.21.15.177:6804/28628 conn(0x7f9ae003e270 0x7f9ae00406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.079+0000 7f9af27fc640 10 monclient: get_auth_request con 0x7f9ae003e270 auth_method 0 2024-09-15T03:58:45.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.081+0000 7f9af2ffd640 1 --2- 172.21.15.110:0/3663603483 >> v2:172.21.15.177:6812/28201 conn(0x7f9ae0038750 0x7f9ae003ac00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4007880 tx=0x7f9ae4004960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.082+0000 7fbc50ab9640 1 --2- 172.21.15.110:0/2637460483 >> v2:172.21.15.177:6812/28201 conn(0x7fbc38038700 0x7fbc3803abb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009d20 tx=0x7fbc400090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.082+0000 7f9adaffd640 1 -- 172.21.15.110:0/3663603483 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9aec026660 con 0x7f9af41369e0 2024-09-15T03:58:45.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.082+0000 7f9adaffd640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-15T03:58:45.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.082+0000 7fbc48ff9640 1 -- 172.21.15.110:0/2637460483 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc4403a370 con 0x7fbc4c09f800 2024-09-15T03:58:45.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.082+0000 7fbc48ff9640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-15T03:58:45.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.083+0000 7fbc512ba640 1 --2- 172.21.15.110:0/2637460483 >> v2:172.21.15.177:6808/28630 conn(0x7fbc3803e1a0 0x7fbc380405d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.084+0000 7f9af27fc640 1 --2- 172.21.15.110:0/3663603483 >> v2:172.21.15.177:6804/28628 conn(0x7f9ae003e270 0x7f9ae00406a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.084+0000 7fbc48ff9640 1 -- 172.21.15.110:0/2637460483 <== osd.5 v2:172.21.15.177:6808/28630 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7fbc38040c60 con 0x7fbc3803e1a0 2024-09-15T03:58:45.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.084+0000 7f9adaffd640 1 -- 172.21.15.110:0/3663603483 <== osd.3 v2:172.21.15.177:6804/28628 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f9ae0040d30 con 0x7f9ae003e270 2024-09-15T03:58:45.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.099+0000 7f78a6b9d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:45.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.099+0000 7f78a6b9d640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.099+0000 7f78a6b9d640 10 monclient: monmap: 2024-09-15T03:58:45.124 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.124 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.124 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:45.100320+0000 2024-09-15T03:58:45.124 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:45.100320+0000 2024-09-15T03:58:45.124 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.124 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.124 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:45.124 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:45.125 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:45.125 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:45.125 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:45.125 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.100+0000 7f78a6b9d640 1 Processor -- start 2024-09-15T03:58:45.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.100+0000 7f78a6b9d640 1 -- start start 2024-09-15T03:58:45.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.100+0000 7f78a6b9d640 10 monclient: init 2024-09-15T03:58:45.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.100+0000 7f78a6b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.100+0000 7f78a6b9d640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-15T03:58:45.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.100+0000 7f78a6b9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78a00a1a10 0x7f78a0131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.100+0000 7f78a6b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f78a00a1a10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.100+0000 7f78a6b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78a009e770 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:45.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.100+0000 7f78a6b9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f78a01369e0 0x7f78a0138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.100+0000 7f78a6b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f78a01369e0 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:45.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.100+0000 7f78a6b9d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a009f290 con 0x7f78a009e770 2024-09-15T03:58:45.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a6b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a6b9d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a00a0850 con 0x7f78a01369e0 2024-09-15T03:58:45.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a6b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a6b9d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a009c9a0 con 0x7f78a00a1a10 2024-09-15T03:58:45.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a6b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a6b9d640 10 monclient(hunting): authenticate will time out at 748.745049s 2024-09-15T03:58:45.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a4939640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78a00a1a10 0x7f78a0131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f78a01369e0 0x7f78a0138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f78a01369e0 0x7f78a0138dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56426/0 (socket says 172.21.15.110:56426) 2024-09-15T03:58:45.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a4939640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78a00a1a10 0x7f78a0131b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54926/0 (socket says 172.21.15.110:54926) 2024-09-15T03:58:45.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 1 -- 172.21.15.110:0/391669901 learned_addr learned my addr 172.21.15.110:0/391669901 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 10 monclient(hunting): get_auth_request con 0x7f78a01369e0 auth_method 0 2024-09-15T03:58:45.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a4939640 10 monclient(hunting): get_auth_request con 0x7f78a00a1a10 auth_method 0 2024-09-15T03:58:45.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a4939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a4939640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a4939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a4939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a4939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a4939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7fe84fb1d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:45.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7fe84fb1d640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7fe84fb1d640 10 monclient: monmap: 2024-09-15T03:58:45.128 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.128 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.128 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:45.102132+0000 2024-09-15T03:58:45.128 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:45.102132+0000 2024-09-15T03:58:45.128 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.128 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.128 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:45.128 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:45.129 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:45.129 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:45.129 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:45.129 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 10 monclient(hunting): handle_auth_done global_id 4230 payload 274 2024-09-15T03:58:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 1 -- 172.21.15.110:0/391669901 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78a00a1a10 msgr2=0x7f78a0131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 1 --2- 172.21.15.110:0/391669901 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78a00a1a10 0x7f78a0131b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 1 -- 172.21.15.110:0/391669901 >> v1:172.21.15.177:6789/0 conn(0x7f78a009e770 legacy=0x7f78a009eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 1 monclient: found mon.noname-a 2024-09-15T03:58:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.101+0000 7f78a513a640 1 -- 172.21.15.110:0/391669901 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a009cc80 con 0x7f78a01369e0 2024-09-15T03:58:45.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.102+0000 7fe84fb1d640 1 Processor -- start 2024-09-15T03:58:45.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.102+0000 7fe84fb1d640 1 -- start start 2024-09-15T03:58:45.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.102+0000 7fe84fb1d640 10 monclient: init 2024-09-15T03:58:45.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.102+0000 7f8148754640 10 monclient: get_monmap_and_config 2024-09-15T03:58:45.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.102+0000 7f8148754640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.102+0000 7f8148754640 10 monclient: monmap: 2024-09-15T03:58:45.132 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.132 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.132 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:45.103603+0000 2024-09-15T03:58:45.132 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:45.103603+0000 2024-09-15T03:58:45.132 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.132 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.132 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:45.132 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:45.132 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:45.132 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:45.132 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:45.132 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84fb1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84fb1d640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-15T03:58:45.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84fb1d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe8480a0290 0x7fe84806a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84fb1d640 10 monclient(hunting): picked mon.noname-c con 0x7fe8480a0290 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:45.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84fb1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe84806e340 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:45.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84fb1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8481335d0 0x7fe8481359b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84fb1d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8481335d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84fb1d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84806ce10 con 0x7fe84806e340 2024-09-15T03:58:45.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84fb1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84fb1d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe84806c570 con 0x7fe8480a0290 2024-09-15T03:58:45.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84fb1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84fb1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84806cb30 con 0x7fe8481335d0 2024-09-15T03:58:45.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84fb1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84fb1d640 10 monclient(hunting): authenticate will time out at 748.747199s 2024-09-15T03:58:45.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7f8148754640 1 Processor -- start 2024-09-15T03:58:45.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7f8148754640 1 -- start start 2024-09-15T03:58:45.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7f8148754640 10 monclient: init 2024-09-15T03:58:45.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8481335d0 0x7fe8481359b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8481335d0 0x7fe8481359b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54938/0 (socket says 172.21.15.110:54938) 2024-09-15T03:58:45.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 1 -- 172.21.15.110:0/4150449305 learned_addr learned my addr 172.21.15.110:0/4150449305 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 10 monclient(hunting): get_auth_request con 0x7fe8481335d0 auth_method 0 2024-09-15T03:58:45.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84d8b9640 1 --2- 172.21.15.110:0/4150449305 >> v2:172.21.15.177:3300/0 conn(0x7fe8480a0290 0x7fe84806a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84d8b9640 10 monclient(hunting): get_auth_request con 0x7fe8480a0290 auth_method 0 2024-09-15T03:58:45.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84d8b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84d8b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84d8b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84d8b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84d8b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84d8b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2024-09-15T03:58:45.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 1 -- 172.21.15.110:0/4150449305 >> v2:172.21.15.177:3300/0 conn(0x7fe8480a0290 msgr2=0x7fe84806a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 1 --2- 172.21.15.110:0/4150449305 >> v2:172.21.15.177:3300/0 conn(0x7fe8480a0290 0x7fe84806a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 1 -- 172.21.15.110:0/4150449305 >> v1:172.21.15.177:6789/0 conn(0x7fe84806e340 legacy=0x7fe84806e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 1 monclient: found mon.noname-b 2024-09-15T03:58:45.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.103+0000 7fe84e0ba640 1 -- 172.21.15.110:0/4150449305 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8480a0660 con 0x7fe8481335d0 2024-09-15T03:58:45.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f8148754640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f8148754640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T03:58:45.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f8148754640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8140067130 0x7f814006eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f8148754640 10 monclient(hunting): picked mon.noname-b con 0x7f8140067130 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f8148754640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f814009c550 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:45.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f8148754640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f814009d0b0 0x7f81401393a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f8148754640 10 monclient(hunting): picked mon.noname-a con 0x7f814009d0b0 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:45.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f8148754640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8140067ec0 con 0x7f814009c550 2024-09-15T03:58:45.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f8148754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f8148754640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f814009b730 con 0x7f814009d0b0 2024-09-15T03:58:45.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f8148754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f8148754640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f814009a830 con 0x7f8140067130 2024-09-15T03:58:45.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f8148754640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f8148754640 10 monclient(hunting): authenticate will time out at 748.748234s 2024-09-15T03:58:45.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8140067130 0x7f814006eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8140067130 0x7f814006eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54946/0 (socket says 172.21.15.110:54946) 2024-09-15T03:58:45.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 1 -- 172.21.15.110:0/4008498545 learned_addr learned my addr 172.21.15.110:0/4008498545 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 10 monclient(hunting): get_auth_request con 0x7f8140067130 auth_method 0 2024-09-15T03:58:45.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2024-09-15T03:58:45.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 1 -- 172.21.15.110:0/4008498545 >> v2:172.21.15.110:3300/0 conn(0x7f814009d0b0 msgr2=0x7f81401393a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 1 --2- 172.21.15.110:0/4008498545 >> v2:172.21.15.110:3300/0 conn(0x7f814009d0b0 0x7f81401393a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 1 -- 172.21.15.110:0/4008498545 >> v1:172.21.15.110:6789/0 conn(0x7f814009c550 legacy=0x7f814009c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 1 monclient: found mon.noname-b 2024-09-15T03:58:45.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f81464f0640 1 -- 172.21.15.110:0/4008498545 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8140067500 con 0x7f8140067130 2024-09-15T03:58:45.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7fed1691d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:45.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7fed1691d640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7fed1691d640 10 monclient: monmap: 2024-09-15T03:58:45.143 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.143 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.143 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:45.105852+0000 2024-09-15T03:58:45.143 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:45.105852+0000 2024-09-15T03:58:45.143 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.143 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.144 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:45.144 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:45.144 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:45.144 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:45.144 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:45.144 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f78a513a640 1 -- 172.21.15.110:0/391669901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a01369e0 msgr2=0x7f78a0138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.104+0000 7f78a513a640 1 --2- 172.21.15.110:0/391669901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a01369e0 0x7f78a0138dc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7898009f20 tx=0x7f7898009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28aa410ee0822859 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f789f7fe640 1 -- 172.21.15.110:0/391669901 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7898007610 con 0x7f78a01369e0 2024-09-15T03:58:45.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f789f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f789f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:45.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f789f7fe640 10 monclient: dump: 2024-09-15T03:58:45.146 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.146 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.146 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.146 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.146 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.146 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.146 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.146 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.146 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.146 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f789f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f789f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f789f7fe640 1 -- 172.21.15.110:0/391669901 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789802d050 con 0x7f78a01369e0 2024-09-15T03:58:45.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f789f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f789f7fe640 1 -- 172.21.15.110:0/391669901 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78980079a0 con 0x7f78a01369e0 2024-09-15T03:58:45.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f789f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f789f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:45.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f789f7fe640 10 monclient: dump: 2024-09-15T03:58:45.147 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.147 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.147 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.147 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.147 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.147 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.147 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.147 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.147 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.147 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f78a6b9d640 5 monclient: authenticate success, global_id 4230 2024-09-15T03:58:45.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f78a6b9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:45.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7f78a6b9d640 4 set_mon_vals no callback set 2024-09-15T03:58:45.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7fed1691d640 1 Processor -- start 2024-09-15T03:58:45.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7fed1691d640 1 -- start start 2024-09-15T03:58:45.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.105+0000 7fed1691d640 10 monclient: init 2024-09-15T03:58:45.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7f78a6b9d640 10 monclient: shutdown 2024-09-15T03:58:45.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7f78a6b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7f78a6b9d640 1 -- 172.21.15.110:0/391669901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a01369e0 msgr2=0x7f78a0138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7f78a6b9d640 1 --2- 172.21.15.110:0/391669901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a01369e0 0x7f78a0138dc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7898009f20 tx=0x7f7898009840 comp rx=0 tx=0).stop 2024-09-15T03:58:45.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7f78a6b9d640 1 -- 172.21.15.110:0/391669901 shutdown_connections 2024-09-15T03:58:45.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7f78a6b9d640 1 --2- 172.21.15.110:0/391669901 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a01369e0 0x7f78a0138dc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7f78a6b9d640 1 --2- 172.21.15.110:0/391669901 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78a00a1a10 0x7f78a0131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7f78a6b9d640 1 -- 172.21.15.110:0/391669901 >> 172.21.15.110:0/391669901 conn(0x7f78a006e860 msgr2=0x7f78a006ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7f78a6b9d640 1 -- 172.21.15.110:0/391669901 shutdown_connections 2024-09-15T03:58:45.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7f78a6b9d640 1 -- 172.21.15.110:0/391669901 wait complete. 2024-09-15T03:58:45.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fed1691d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fed1691d640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T03:58:45.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fed1691d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed100a1a10 0x7fed10131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7f78a6b9d640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7f78a6b9d640 10 monclient: monmap: 2024-09-15T03:58:45.151 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.151 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:45.151 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:45.151 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:45.151 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.151 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.151 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:45.151 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:45.151 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:45.152 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fed1691d640 10 monclient(hunting): picked mon.noname-b con 0x7fed100a1a10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fe84e0ba640 1 -- 172.21.15.110:0/4150449305 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8481335d0 msgr2=0x7fe8481359b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fe84e0ba640 1 --2- 172.21.15.110:0/4150449305 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8481335d0 0x7fe8481359b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe84000b1f0 tx=0x7fe840009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e422759194232cb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fed1691d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed1009e770 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:45.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fed1691d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed101369e0 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:45.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84c8b7640 1 -- 172.21.15.110:0/4150449305 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe840032470 con 0x7fe8481335d0 2024-09-15T03:58:45.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84c8b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84c8b7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:45.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84c8b7640 10 monclient: dump: 2024-09-15T03:58:45.153 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.154 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.154 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.154 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.154 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.154 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.154 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.154 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.154 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.154 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84c8b7640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84c8b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84c8b7640 1 -- 172.21.15.110:0/4150449305 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8400325d0 con 0x7fe8481335d0 2024-09-15T03:58:45.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84c8b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84c8b7640 1 -- 172.21.15.110:0/4150449305 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8400329b0 con 0x7fe8481335d0 2024-09-15T03:58:45.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84c8b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84c8b7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:45.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84c8b7640 10 monclient: dump: 2024-09-15T03:58:45.155 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.155 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.155 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.155 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.155 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.155 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.155 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.155 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.155 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.156 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7f78a6b9d640 1 Processor -- start 2024-09-15T03:58:45.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fed1691d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1009f290 con 0x7fed1009e770 2024-09-15T03:58:45.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fed1691d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed100a0850 con 0x7fed101369e0 2024-09-15T03:58:45.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fed0f7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fed1009e770 0x7fed1009eb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33766/0 (socket says 172.21.15.110:33766) 2024-09-15T03:58:45.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fed0f7fe640 1 -- 172.21.15.110:0/3964877954 learned_addr learned my addr 172.21.15.110:0/3964877954 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:45.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7f78a6b9d640 1 -- start start 2024-09-15T03:58:45.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fed1691d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fed1691d640 1 -- 172.21.15.110:0/3964877954 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1009c9a0 con 0x7fed100a1a10 2024-09-15T03:58:45.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fed1691d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.106+0000 7fed1691d640 10 monclient(hunting): authenticate will time out at 748.750879s 2024-09-15T03:58:45.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 1 --2- 172.21.15.110:0/3964877954 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed100a1a10 0x7fed10131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 10 monclient(hunting): get_auth_request con 0x7fed100a1a10 auth_method 0 2024-09-15T03:58:45.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84fb1d640 5 monclient: authenticate success, global_id 4235 2024-09-15T03:58:45.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84fb1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:45.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fe84fb1d640 4 set_mon_vals no callback set 2024-09-15T03:58:45.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7f78a6b9d640 10 monclient: init 2024-09-15T03:58:45.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-09-15T03:58:45.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7f81464f0640 1 -- 172.21.15.110:0/4008498545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8140067130 msgr2=0x7f814006eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7f81464f0640 1 --2- 172.21.15.110:0/4008498545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8140067130 0x7f814006eb80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f813c009910 tx=0x7f813c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33486694df6da4a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 1 -- 172.21.15.110:0/3964877954 >> v1:172.21.15.177:6789/0 conn(0x7fed101369e0 legacy=0x7fed10138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:58:45.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78a009e770 0x7f78a006a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f81454ee640 1 -- 172.21.15.110:0/4008498545 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c032440 con 0x7f8140067130 2024-09-15T03:58:45.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f81454ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f81454ee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:45.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f81454ee640 10 monclient: dump: 2024-09-15T03:58:45.161 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.161 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.161 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.161 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.161 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.161 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.162 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.162 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.162 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.162 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f81454ee640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f81454ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f81454ee640 1 -- 172.21.15.110:0/4008498545 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813c0325a0 con 0x7f8140067130 2024-09-15T03:58:45.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f81454ee640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f81454ee640 1 -- 172.21.15.110:0/4008498545 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c032940 con 0x7f8140067130 2024-09-15T03:58:45.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f81454ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f81454ee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:45.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f81454ee640 10 monclient: dump: 2024-09-15T03:58:45.162 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.162 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.163 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.163 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.163 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.163 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.163 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.163 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.163 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.163 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f8148754640 5 monclient: authenticate success, global_id 4238 2024-09-15T03:58:45.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f8148754640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:45.163 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f8148754640 4 set_mon_vals no callback set 2024-09-15T03:58:45.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f78a009e770 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78a009e770 0x7f78a006a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78a009e770 0x7f78a006a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:54954/0 (socket says 172.21.15.110:54954) 2024-09-15T03:58:45.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7fe84fb1d640 10 monclient: shutdown 2024-09-15T03:58:45.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7fe84fb1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78a00a1a10 0x7f78a006ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f78a00a1a10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:45.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a01369e0 0x7f78a006b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f78a01369e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:45.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00a4b10 con 0x7f78a01369e0 2024-09-15T03:58:45.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a006dd80 con 0x7f78a00a1a10 2024-09-15T03:58:45.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a006c020 con 0x7f78a009e770 2024-09-15T03:58:45.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a6b9d640 10 monclient(hunting): authenticate will time out at 748.752474s 2024-09-15T03:58:45.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 1 -- 172.21.15.110:0/966291440 learned_addr learned my addr 172.21.15.110:0/966291440 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 10 monclient(hunting): get_auth_request con 0x7f78a009e770 auth_method 0 2024-09-15T03:58:45.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7fe84fb1d640 1 -- 172.21.15.110:0/4150449305 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8481335d0 msgr2=0x7fe8481359b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.167 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7fe84fb1d640 1 --2- 172.21.15.110:0/4150449305 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8481335d0 0x7fe8481359b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe84000b1f0 tx=0x7fe840009840 comp rx=0 tx=0).stop 2024-09-15T03:58:45.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 10 monclient(hunting): handle_auth_done global_id 4244 payload 824 2024-09-15T03:58:45.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 1 -- 172.21.15.110:0/966291440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78a00a1a10 msgr2=0x7f78a006ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T03:58:45.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 1 --2- 172.21.15.110:0/966291440 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78a00a1a10 0x7f78a006ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 1 -- 172.21.15.110:0/966291440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a01369e0 msgr2=0x7f78a006b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 1 --2- 172.21.15.110:0/966291440 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78a01369e0 0x7f78a006b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.168 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 1 monclient: found mon.noname-c 2024-09-15T03:58:45.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7f78a4939640 1 -- 172.21.15.110:0/966291440 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7898009ae0 con 0x7f78a009e770 2024-09-15T03:58:45.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 1 -- 172.21.15.110:0/3964877954 >> v1:172.21.15.110:6789/0 conn(0x7fed1009e770 legacy=0x7fed1009eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.169 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7fe84fb1d640 1 -- 172.21.15.110:0/4150449305 shutdown_connections 2024-09-15T03:58:45.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 1 monclient: found mon.noname-b 2024-09-15T03:58:45.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.107+0000 7fed0ffff640 1 -- 172.21.15.110:0/3964877954 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1009cc80 con 0x7fed100a1a10 2024-09-15T03:58:45.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7fe84fb1d640 1 --2- 172.21.15.110:0/4150449305 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8481335d0 0x7fe8481359b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7fe84fb1d640 1 --2- 172.21.15.110:0/4150449305 >> v2:172.21.15.177:3300/0 conn(0x7fe8480a0290 0x7fe84806a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.170 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.108+0000 7fe84fb1d640 1 -- 172.21.15.110:0/4150449305 >> 172.21.15.110:0/4150449305 conn(0x7fe84809d680 msgr2=0x7fe84809da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7f8148754640 10 monclient: shutdown 2024-09-15T03:58:45.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7f8148754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7f8148754640 1 -- 172.21.15.110:0/4008498545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8140067130 msgr2=0x7f814006eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7f8148754640 1 --2- 172.21.15.110:0/4008498545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8140067130 0x7f814006eb80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f813c009910 tx=0x7f813c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7fe84fb1d640 1 -- 172.21.15.110:0/4150449305 shutdown_connections 2024-09-15T03:58:45.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7f8148754640 1 -- 172.21.15.110:0/4008498545 shutdown_connections 2024-09-15T03:58:45.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7fe84fb1d640 1 -- 172.21.15.110:0/4150449305 wait complete. 2024-09-15T03:58:45.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7f8148754640 1 --2- 172.21.15.110:0/4008498545 >> v2:172.21.15.110:3300/0 conn(0x7f814009d0b0 0x7f81401393a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7f8148754640 1 --2- 172.21.15.110:0/4008498545 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8140067130 0x7f814006eb80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7f8148754640 1 -- 172.21.15.110:0/4008498545 >> 172.21.15.110:0/4008498545 conn(0x7f81400a2780 msgr2=0x7f8140068240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7f8148754640 1 -- 172.21.15.110:0/4008498545 shutdown_connections 2024-09-15T03:58:45.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7f8148754640 1 -- 172.21.15.110:0/4008498545 wait complete. 2024-09-15T03:58:45.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7f8148754640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7f8148754640 10 monclient: monmap: 2024-09-15T03:58:45.173 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.173 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:45.173 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:45.173 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:45.173 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.174 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.174 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:45.174 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:45.174 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:45.174 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7f9af9709640 1 -- 172.21.15.110:0/3663603483 --> v2:172.21.15.177:6804/28628 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9af4098d30 con 0x7f9ae003e270 2024-09-15T03:58:45.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7fe84fb1d640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7fe84fb1d640 10 monclient: monmap: 2024-09-15T03:58:45.175 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.175 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:45.175 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:45.175 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:45.175 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.175 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.175 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:45.175 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:45.175 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:45.175 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.109+0000 7fbc52d1d640 1 -- 172.21.15.110:0/2637460483 --> v2:172.21.15.177:6808/28630 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbc2c002b40 con 0x7fbc3803e1a0 2024-09-15T03:58:45.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 1 Processor -- start 2024-09-15T03:58:45.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 1 -- start start 2024-09-15T03:58:45.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 10 monclient: init 2024-09-15T03:58:45.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f9adaffd640 1 -- 172.21.15.110:0/3663603483 <== osd.3 v2:172.21.15.177:6804/28628 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9af4098d30 con 0x7f9ae003e270 2024-09-15T03:58:45.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 1 Processor -- start 2024-09-15T03:58:45.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fbc48ff9640 1 -- 172.21.15.110:0/2637460483 <== osd.5 v2:172.21.15.177:6808/28630 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbc2c002b40 con 0x7fbc3803e1a0 2024-09-15T03:58:45.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 1 -- start start 2024-09-15T03:58:45.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 10 monclient: init 2024-09-15T03:58:45.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f9af9709640 1 -- 172.21.15.110:0/3663603483 >> v2:172.21.15.177:6804/28628 conn(0x7f9ae003e270 msgr2=0x7f9ae00406a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f9af9709640 1 --2- 172.21.15.110:0/3663603483 >> v2:172.21.15.177:6804/28628 conn(0x7f9ae003e270 0x7f9ae00406a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:58:45.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140067130 0x7f8140140d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 10 monclient(hunting): picked mon.noname-a con 0x7f8140067130 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:45.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fbc52d1d640 1 -- 172.21.15.110:0/2637460483 >> v2:172.21.15.177:6808/28630 conn(0x7fbc3803e1a0 msgr2=0x7fbc380405d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fbc52d1d640 1 --2- 172.21.15.110:0/2637460483 >> v2:172.21.15.177:6808/28630 conn(0x7fbc3803e1a0 0x7fbc380405d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f814009c550 0x7f81401316e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 10 monclient(hunting): picked mon.noname-c con 0x7f814009c550 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f814009d0b0 0x7f814011b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 10 monclient(hunting): picked mon.noname-b con 0x7f814009d0b0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:45.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814009e250 con 0x7f8140067130 2024-09-15T03:58:45.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f81464f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140067130 0x7f8140140d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f9af9709640 1 -- 172.21.15.110:0/3663603483 >> v2:172.21.15.177:6812/28201 conn(0x7f9ae0038750 msgr2=0x7f9ae003ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814006da90 con 0x7f814009d0b0 2024-09-15T03:58:45.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8140141bd0 con 0x7f814009c550 2024-09-15T03:58:45.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f8148754640 10 monclient(hunting): authenticate will time out at 748.754747s 2024-09-15T03:58:45.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f81464f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140067130 0x7f8140140d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56454/0 (socket says 172.21.15.110:56454) 2024-09-15T03:58:45.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f9af9709640 1 --2- 172.21.15.110:0/3663603483 >> v2:172.21.15.177:6812/28201 conn(0x7f9ae0038750 0x7f9ae003ac00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4007880 tx=0x7f9ae4004960 comp rx=0 tx=0).stop 2024-09-15T03:58:45.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:58:45.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f9af9709640 10 monclient: shutdown 2024-09-15T03:58:45.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f81464f0640 1 -- 172.21.15.110:0/2842859561 learned_addr learned my addr 172.21.15.110:0/2842859561 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f81464f0640 10 monclient(hunting): get_auth_request con 0x7f8140067130 auth_method 0 2024-09-15T03:58:45.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f81464f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f81464f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f81464f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f9af9709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fbc52d1d640 1 -- 172.21.15.110:0/2637460483 >> v2:172.21.15.177:6812/28201 conn(0x7fbc38038700 msgr2=0x7fbc3803abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fbc52d1d640 1 --2- 172.21.15.110:0/2637460483 >> v2:172.21.15.177:6812/28201 conn(0x7fbc38038700 0x7fbc3803abb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009d20 tx=0x7fbc400090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fbc52d1d640 10 monclient: shutdown 2024-09-15T03:58:45.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fbc52d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fbc52d1d640 1 -- 172.21.15.110:0/2637460483 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09f800 msgr2=0x7fbc4c06bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f81464f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f81464f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f81464f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe84806e340 0x7fe848099fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 10 monclient(hunting): picked mon.noname-a con 0x7fe84806e340 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:45.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8480a0290 0x7fe848117050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8480a0290 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:45.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f9af9709640 1 -- 172.21.15.110:0/3663603483 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9af41369e0 msgr2=0x7f9af4097f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fbc52d1d640 1 --2- 172.21.15.110:0/2637460483 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fbc4c09f800 0x7fbc4c06bf90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbc4402ac70 tx=0x7fbc44036000 comp rx=0 tx=0).stop 2024-09-15T03:58:45.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fbc512ba640 1 -- 172.21.15.110:0/2637460483 reap_dead start 2024-09-15T03:58:45.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8481335d0 0x7fe848117590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 10 monclient(hunting): picked mon.noname-c con 0x7fe8481335d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe848136860 con 0x7fe84806e340 2024-09-15T03:58:45.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84811a210 con 0x7fe8480a0290 2024-09-15T03:58:45.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8481186a0 con 0x7fe8481335d0 2024-09-15T03:58:45.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fe84fb1d640 10 monclient(hunting): authenticate will time out at 748.754928s 2024-09-15T03:58:45.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f9af9709640 1 --2- 172.21.15.110:0/3663603483 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9af41369e0 0x7f9af4097f00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00cf30 tx=0x7f9aec007840 comp rx=0 tx=0).stop 2024-09-15T03:58:45.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fbc52d1d640 1 -- 172.21.15.110:0/2637460483 shutdown_connections 2024-09-15T03:58:45.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7fbc52d1d640 1 -- 172.21.15.110:0/2637460483 >> 172.21.15.110:0/2637460483 conn(0x7fbc4c06e860 msgr2=0x7fbc4c0a4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.110+0000 7f9af37fe640 1 -- 172.21.15.110:0/3663603483 reap_dead start 2024-09-15T03:58:45.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f8146cf1640 1 --2- 172.21.15.110:0/2842859561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f814009d0b0 0x7f814011b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f81464f0640 10 monclient(hunting): handle_auth_done global_id 4233 payload 824 2024-09-15T03:58:45.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f81464f0640 1 -- 172.21.15.110:0/2842859561 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f814009c550 msgr2=0x7f81401316e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f81464f0640 1 --2- 172.21.15.110:0/2842859561 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f814009c550 0x7f81401316e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f9af9709640 1 -- 172.21.15.110:0/3663603483 shutdown_connections 2024-09-15T03:58:45.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f9af9709640 1 -- 172.21.15.110:0/3663603483 >> 172.21.15.110:0/3663603483 conn(0x7f9af406e860 msgr2=0x7f9af40a3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f81464f0640 1 -- 172.21.15.110:0/2842859561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f814009d0b0 msgr2=0x7f814011b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f81464f0640 1 --2- 172.21.15.110:0/2842859561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f814009d0b0 0x7f814011b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f81464f0640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f81464f0640 1 monclient: found mon.noname-a 2024-09-15T03:58:45.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f81464f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f81464f0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f81464f0640 1 -- 172.21.15.110:0/2842859561 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c009450 con 0x7f8140067130 2024-09-15T03:58:45.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f8146cf1640 10 monclient: get_auth_request con 0x7f814009d0b0 auth_method 0 2024-09-15T03:58:45.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f8146cf1640 1 --2- 172.21.15.110:0/2842859561 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f814009d0b0 0x7f814011b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:58:45.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8480a0290 0x7fe848117050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8480a0290 0x7fe848117050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.110:59752/0 (socket says 172.21.15.110:59752) 2024-09-15T03:58:45.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 1 -- 172.21.15.110:0/2390232608 learned_addr learned my addr 172.21.15.110:0/2390232608 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 10 monclient(hunting): get_auth_request con 0x7fe8480a0290 auth_method 0 2024-09-15T03:58:45.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f9af9709640 1 -- 172.21.15.110:0/3663603483 shutdown_connections 2024-09-15T03:58:45.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f9af9709640 1 -- 172.21.15.110:0/3663603483 wait complete. 2024-09-15T03:58:45.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fbc52d1d640 1 -- 172.21.15.110:0/2637460483 shutdown_connections 2024-09-15T03:58:45.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f78a4939640 1 -- 172.21.15.110:0/966291440 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78a009e770 msgr2=0x7f78a006a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f78a4939640 1 --2- 172.21.15.110:0/966291440 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78a009e770 0x7f78a006a720 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7894017480 tx=0x7f7894008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed0ffff640 1 -- 172.21.15.110:0/3964877954 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed100a1a10 msgr2=0x7fed10131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed0ffff640 1 --2- 172.21.15.110:0/3964877954 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed100a1a10 0x7fed10131b90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fed00009880 tx=0x7fed000090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=297b44dd11dfacbb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d8b9640 1 --2- 172.21.15.110:0/2390232608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe84806e340 0x7fe848099fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84e0ba640 1 --2- 172.21.15.110:0/2390232608 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8481335d0 0x7fe848117590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.192 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fbc52d1d640 1 -- 172.21.15.110:0/2637460483 wait complete. 2024-09-15T03:58:45.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d8b9640 10 monclient(hunting): get_auth_request con 0x7fe84806e340 auth_method 0 2024-09-15T03:58:45.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed0effd640 1 -- 172.21.15.110:0/3964877954 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00032440 con 0x7fed100a1a10 2024-09-15T03:58:45.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed0effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:45.193 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed0effd640 10 monclient: dump: 2024-09-15T03:58:45.193 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.194 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.194 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.194 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.194 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.194 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.194 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.194 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.194 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.194 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed0effd640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed1691d640 5 monclient: authenticate success, global_id 4241 2024-09-15T03:58:45.194 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed1691d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed0effd640 1 -- 172.21.15.110:0/3964877954 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed000325a0 con 0x7fed100a1a10 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed0effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed0effd640 1 -- 172.21.15.110:0/3964877954 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00032940 con 0x7fed100a1a10 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed0effd640 10 monclient: dump: 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.195 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.196 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed1691d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:45.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fed1691d640 4 set_mon_vals no callback set 2024-09-15T03:58:45.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d8b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d8b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d8b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.196 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84e0ba640 10 monclient(hunting): get_auth_request con 0x7fe8481335d0 auth_method 0 2024-09-15T03:58:45.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f789cff9640 1 -- 172.21.15.110:0/966291440 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7894013e40 con 0x7f78a009e770 2024-09-15T03:58:45.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f789cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f789cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:45.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84e0ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84e0ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84e0ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 10 monclient(hunting): handle_auth_done global_id 4222 payload 824 2024-09-15T03:58:45.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 1 -- 172.21.15.110:0/2390232608 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8481335d0 msgr2=0x7fe848117590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 1 --2- 172.21.15.110:0/2390232608 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe8481335d0 0x7fe848117590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.197 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 1 -- 172.21.15.110:0/2390232608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe84806e340 msgr2=0x7fe848099fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f789cff9640 10 monclient: dump: 2024-09-15T03:58:45.198 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.198 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.198 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.198 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.198 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.198 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.198 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.198 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.198 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.198 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f789cff9640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f789cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f789cff9640 1 -- 172.21.15.110:0/966291440 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7894019040 con 0x7f78a009e770 2024-09-15T03:58:45.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f789cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 1 --2- 172.21.15.110:0/2390232608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe84806e340 0x7fe848099fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 1 monclient: found mon.noname-b 2024-09-15T03:58:45.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-09-15T03:58:45.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d0b8640 1 -- 172.21.15.110:0/2390232608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe840009ae0 con 0x7fe8480a0290 2024-09-15T03:58:45.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f789cff9640 1 -- 172.21.15.110:0/966291440 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789401c980 con 0x7f78a009e770 2024-09-15T03:58:45.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f789cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f789cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:45.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f789cff9640 10 monclient: dump: 2024-09-15T03:58:45.200 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.200 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.200 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.200 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.200 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.200 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.201 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.201 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.201 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.201 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f78a6b9d640 5 monclient: authenticate success, global_id 4244 2024-09-15T03:58:45.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f78a6b9d640 10 monclient: _renew_subs 2024-09-15T03:58:45.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f78a6b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f78a6b9d640 1 -- 172.21.15.110:0/966291440 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a006daa0 con 0x7f78a009e770 2024-09-15T03:58:45.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7fe84d8b9640 1 --2- 172.21.15.110:0/2390232608 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe84806e340 0x7fe848099fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:58:45.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f78a6b9d640 10 monclient: _renew_subs 2024-09-15T03:58:45.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f78a6b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f78a6b9d640 1 -- 172.21.15.110:0/966291440 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a006cbb0 con 0x7f78a009e770 2024-09-15T03:58:45.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.111+0000 7f789effd640 4 set_mon_vals no callback set 2024-09-15T03:58:45.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.112+0000 7fed1691d640 10 monclient: shutdown 2024-09-15T03:58:45.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.112+0000 7fed1691d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.112+0000 7fed1691d640 1 -- 172.21.15.110:0/3964877954 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed100a1a10 msgr2=0x7fed10131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.112+0000 7fed1691d640 1 --2- 172.21.15.110:0/3964877954 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed100a1a10 0x7fed10131b90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fed00009880 tx=0x7fed000090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.112+0000 7fed1691d640 1 -- 172.21.15.110:0/3964877954 shutdown_connections 2024-09-15T03:58:45.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.112+0000 7fed1691d640 1 --2- 172.21.15.110:0/3964877954 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed100a1a10 0x7fed10131b90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.112+0000 7fed1691d640 1 -- 172.21.15.110:0/3964877954 >> 172.21.15.110:0/3964877954 conn(0x7fed1006e860 msgr2=0x7fed1006ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.112+0000 7fed1691d640 1 -- 172.21.15.110:0/3964877954 shutdown_connections 2024-09-15T03:58:45.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7f789cff9640 1 -- 172.21.15.110:0/966291440 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f789401ccf0 con 0x7f78a009e770 2024-09-15T03:58:45.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7f789cff9640 1 --2- 172.21.15.110:0/966291440 >> v2:172.21.15.177:6812/28201 conn(0x7f7884038750 0x7f788403ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7f78a6b9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7f78a6b9d640 1 -- 172.21.15.110:0/966291440 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f78a00647e0 con 0x7f78a009e770 2024-09-15T03:58:45.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.112+0000 7fed1691d640 1 -- 172.21.15.110:0/3964877954 wait complete. 2024-09-15T03:58:45.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7f789cff9640 1 -- 172.21.15.110:0/966291440 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f78940541e0 con 0x7f78a009e770 2024-09-15T03:58:45.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7fed1691d640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7fed1691d640 10 monclient: monmap: 2024-09-15T03:58:45.205 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.205 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:45.205 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:45.205 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:45.205 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.205 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.205 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:45.205 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:45.205 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:45.205 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7f789ffff640 1 --2- 172.21.15.110:0/966291440 >> v2:172.21.15.177:6812/28201 conn(0x7f7884038750 0x7f788403ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7f789ffff640 10 monclient: get_auth_request con 0x7f7884038750 auth_method 0 2024-09-15T03:58:45.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7f81464f0640 1 -- 172.21.15.110:0/2842859561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140067130 msgr2=0x7f8140140d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7f81464f0640 1 --2- 172.21.15.110:0/2842859561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140067130 0x7f8140140d50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f813c009210 tx=0x7f813c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7f789cff9640 1 --2- 172.21.15.110:0/966291440 >> v2:172.21.15.110:6804/28368 conn(0x7f788403e270 0x7f78840406a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7f789cff9640 1 -- 172.21.15.110:0/966291440 --> v2:172.21.15.110:6804/28368 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7884040d30 con 0x7f788403e270 2024-09-15T03:58:45.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7f789cff9640 1 -- 172.21.15.110:0/966291440 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7894028d50 con 0x7f78a009e770 2024-09-15T03:58:45.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7f789cff9640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-15T03:58:45.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f81464f0640 1 --2- 172.21.15.110:0/2842859561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140067130 0x7f8140140d50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f813c009210 tx=0x7f813c032000 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f813c03ce30 mon_map magic: 0 v1 2024-09-15T03:58:45.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.113+0000 7fed1691d640 1 Processor -- start 2024-09-15T03:58:45.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f81367fc640 1 -- 172.21.15.110:0/2842859561 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c03c910 con 0x7f8140067130 2024-09-15T03:58:45.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f81367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f81367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:45.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fed1691d640 1 -- start start 2024-09-15T03:58:45.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f81367fc640 10 monclient: dump: 2024-09-15T03:58:45.209 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.209 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.209 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.209 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.209 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.209 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.209 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.209 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.209 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.209 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f81367fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f81367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fed1691d640 10 monclient: init 2024-09-15T03:58:45.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f81367fc640 1 -- 172.21.15.110:0/2842859561 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f813c03ca70 con 0x7f8140067130 2024-09-15T03:58:45.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f81367fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f8148754640 5 monclient: authenticate success, global_id 4233 2024-09-15T03:58:45.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f8148754640 10 monclient: _renew_subs 2024-09-15T03:58:45.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f8148754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f8148754640 1 -- 172.21.15.110:0/2842859561 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f814011cd30 con 0x7f8140067130 2024-09-15T03:58:45.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f8144ced640 4 set_mon_vals no callback set 2024-09-15T03:58:45.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f8148754640 10 monclient: _renew_subs 2024-09-15T03:58:45.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f8148754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f8148754640 1 -- 172.21.15.110:0/2842859561 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f814011d8a0 con 0x7f8140067130 2024-09-15T03:58:45.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fe84d0b8640 1 -- 172.21.15.110:0/2390232608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8480a0290 msgr2=0x7fe848117050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fe84d0b8640 1 --2- 172.21.15.110:0/2390232608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8480a0290 0x7fe848117050 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe84400b5e0 tx=0x7fe8440090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fed1691d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fed1691d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:58:45.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fed1691d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed1009e770 0x7fed1009a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fe83dffb640 1 -- 172.21.15.110:0/2390232608 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844009160 con 0x7fe8480a0290 2024-09-15T03:58:45.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fe83dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fe83dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:45.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fe83dffb640 10 monclient: dump: 2024-09-15T03:58:45.212 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.212 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.212 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.212 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.213 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.213 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.213 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.213 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.213 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.213 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fe83dffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fe83dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fe83dffb640 1 -- 172.21.15.110:0/2390232608 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe844010ad0 con 0x7fe8480a0290 2024-09-15T03:58:45.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fe83dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fe83dffb640 1 -- 172.21.15.110:0/2390232608 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe844010e90 con 0x7fe8480a0290 2024-09-15T03:58:45.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fe83dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fe83dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:45.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7fe83dffb640 10 monclient: dump: 2024-09-15T03:58:45.214 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.214 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.214 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.214 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.214 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.214 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.214 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.214 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.214 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.214 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f78a513a640 1 --2- 172.21.15.110:0/966291440 >> v2:172.21.15.110:6804/28368 conn(0x7f788403e270 0x7f78840406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.114+0000 7f78a513a640 10 monclient: get_auth_request con 0x7f788403e270 auth_method 0 2024-09-15T03:58:45.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fe83ffff640 4 set_mon_vals no callback set 2024-09-15T03:58:45.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fe84fb1d640 5 monclient: authenticate success, global_id 4222 2024-09-15T03:58:45.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fe84fb1d640 10 monclient: _renew_subs 2024-09-15T03:58:45.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fe84fb1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:45.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fe84fb1d640 1 -- 172.21.15.110:0/2390232608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe848119c50 con 0x7fe8480a0290 2024-09-15T03:58:45.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed1691d640 10 monclient(hunting): picked mon.noname-c con 0x7fed1009e770 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed1691d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed100a1a10 0x7fed100962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed1691d640 10 monclient(hunting): picked mon.noname-b con 0x7fed100a1a10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:45.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed1691d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed101369e0 0x7fed100967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed1009e770 0x7fed1009a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed1009e770 0x7fed1009a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55000/0 (socket says 172.21.15.110:55000) 2024-09-15T03:58:45.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed1691d640 10 monclient(hunting): picked mon.noname-a con 0x7fed101369e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:45.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed1691d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed1691d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed100a11c0 con 0x7fed101369e0 2024-09-15T03:58:45.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed1691d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed1691d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed100a3dc0 con 0x7fed100a1a10 2024-09-15T03:58:45.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed1691d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed1691d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed10099b90 con 0x7fed1009e770 2024-09-15T03:58:45.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed1691d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed1691d640 10 monclient(hunting): authenticate will time out at 748.759318s 2024-09-15T03:58:45.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0f7fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed100a1a10 0x7fed100962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0f7fe640 10 monclient(hunting): get_auth_request con 0x7fed100a1a10 auth_method 0 2024-09-15T03:58:45.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 1 -- 172.21.15.110:0/3354529013 learned_addr learned my addr 172.21.15.110:0/3354529013 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 10 monclient(hunting): get_auth_request con 0x7fed1009e770 auth_method 0 2024-09-15T03:58:45.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fe84fb1d640 10 monclient: _renew_subs 2024-09-15T03:58:45.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fe84fb1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:45.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fe84fb1d640 1 -- 172.21.15.110:0/2390232608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe848132180 con 0x7fe8480a0290 2024-09-15T03:58:45.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-09-15T03:58:45.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 1 -- 172.21.15.110:0/3354529013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed100a1a10 msgr2=0x7fed100962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7f81367fc640 1 -- 172.21.15.110:0/2842859561 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f813c03ce30 con 0x7f8140067130 2024-09-15T03:58:45.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7f81367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7f81367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:45.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7f81367fc640 10 monclient: dump: 2024-09-15T03:58:45.221 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.221 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.221 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.221 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.221 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.221 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.221 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.221 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.221 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.222 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7f81367fc640 1 -- 172.21.15.110:0/2842859561 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f813c039070 con 0x7f8140067130 2024-09-15T03:58:45.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7f8148754640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7f8148754640 1 -- 172.21.15.110:0/2842859561 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f814011e8e0 con 0x7f8140067130 2024-09-15T03:58:45.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7f81367fc640 1 --2- 172.21.15.110:0/2842859561 >> v2:172.21.15.177:6812/28201 conn(0x7f8124038750 0x7f812403ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 1 --2- 172.21.15.110:0/3354529013 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fed100a1a10 0x7fed100962a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 1 -- 172.21.15.110:0/3354529013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed101369e0 msgr2=0x7fed100967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 1 --2- 172.21.15.110:0/3354529013 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fed101369e0 0x7fed100967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 1 monclient: found mon.noname-c 2024-09-15T03:58:45.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.115+0000 7fed0ffff640 1 -- 172.21.15.110:0/3354529013 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed00009390 con 0x7fed1009e770 2024-09-15T03:58:45.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.116+0000 7f789ffff640 1 --2- 172.21.15.110:0/966291440 >> v2:172.21.15.177:6812/28201 conn(0x7f7884038750 0x7f788403ac00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f788c007fd0 tx=0x7f788c007a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.116+0000 7f81367fc640 1 -- 172.21.15.110:0/2842859561 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f813c06be40 con 0x7f8140067130 2024-09-15T03:58:45.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.116+0000 7f81367fc640 1 --2- 172.21.15.110:0/2842859561 >> v2:172.21.15.110:6808/28367 conn(0x7f812403e1f0 0x7f8124040620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.116+0000 7f81367fc640 1 -- 172.21.15.110:0/2842859561 --> v2:172.21.15.110:6808/28367 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8124040cb0 con 0x7f812403e1f0 2024-09-15T03:58:45.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.116+0000 7fe84fb1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:45.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.116+0000 7f8145cef640 1 --2- 172.21.15.110:0/2842859561 >> v2:172.21.15.177:6812/28201 conn(0x7f8124038750 0x7f812403ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.116+0000 7fe84fb1d640 1 -- 172.21.15.110:0/2390232608 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe8480647e0 con 0x7fe8480a0290 2024-09-15T03:58:45.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.116+0000 7f8145cef640 10 monclient: get_auth_request con 0x7f8124038750 auth_method 0 2024-09-15T03:58:45.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.116+0000 7f8146cf1640 1 --2- 172.21.15.110:0/2842859561 >> v2:172.21.15.110:6808/28367 conn(0x7f812403e1f0 0x7f8124040620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.116+0000 7f8146cf1640 10 monclient: get_auth_request con 0x7f812403e1f0 auth_method 0 2024-09-15T03:58:45.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.116+0000 7fe83dffb640 1 -- 172.21.15.110:0/2390232608 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fe844016070 con 0x7fe8480a0290 2024-09-15T03:58:45.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.117+0000 7fe83dffb640 1 --2- 172.21.15.110:0/2390232608 >> v2:172.21.15.177:6812/28201 conn(0x7fe82c038750 0x7fe82c03ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.117+0000 7fe83dffb640 1 -- 172.21.15.110:0/2390232608 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7fe84404efe0 con 0x7fe8480a0290 2024-09-15T03:58:45.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.117+0000 7fe84d8b9640 1 --2- 172.21.15.110:0/2390232608 >> v2:172.21.15.177:6812/28201 conn(0x7fe82c038750 0x7fe82c03ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.117+0000 7fe84d8b9640 10 monclient: get_auth_request con 0x7fe82c038750 auth_method 0 2024-09-15T03:58:45.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.117+0000 7fe83dffb640 1 --2- 172.21.15.110:0/2390232608 >> v2:172.21.15.110:6800/28369 conn(0x7fe82c03e180 0x7fe82c0405b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.117+0000 7fe83dffb640 1 -- 172.21.15.110:0/2390232608 --> v2:172.21.15.110:6800/28369 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe82c040c40 con 0x7fe82c03e180 2024-09-15T03:58:45.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.117+0000 7fe83dffb640 1 -- 172.21.15.110:0/2390232608 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe844051570 con 0x7fe8480a0290 2024-09-15T03:58:45.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.117+0000 7fe83dffb640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-15T03:58:45.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.117+0000 7fe84e0ba640 1 --2- 172.21.15.110:0/2390232608 >> v2:172.21.15.110:6800/28369 conn(0x7fe82c03e180 0x7fe82c0405b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.117+0000 7fe84e0ba640 10 monclient: get_auth_request con 0x7fe82c03e180 auth_method 0 2024-09-15T03:58:45.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.118+0000 7fed0ffff640 1 -- 172.21.15.110:0/3354529013 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed1009e770 msgr2=0x7fed1009a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.118+0000 7fed0ffff640 1 --2- 172.21.15.110:0/3354529013 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed1009e770 0x7fed1009a540 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fed00009880 tx=0x7fed00032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.118+0000 7fecf7fff640 1 -- 172.21.15.110:0/3354529013 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:58:45.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.119+0000 7f78a513a640 1 --2- 172.21.15.110:0/966291440 >> v2:172.21.15.110:6804/28368 conn(0x7f788403e270 0x7f78840406a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.119+0000 7f8145cef640 1 --2- 172.21.15.110:0/2842859561 >> v2:172.21.15.177:6812/28201 conn(0x7f8124038750 0x7f812403ac00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8140068b30 tx=0x7f8130006c50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.119+0000 7f789cff9640 1 -- 172.21.15.110:0/966291440 <== osd.2 v2:172.21.15.110:6804/28368 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f7884040d30 con 0x7f788403e270 2024-09-15T03:58:45.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.119+0000 7fe84d8b9640 1 --2- 172.21.15.110:0/2390232608 >> v2:172.21.15.177:6812/28201 conn(0x7fe82c038750 0x7fe82c03ac00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe838009cf0 tx=0x7fe838008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fecf7fff640 1 -- 172.21.15.110:0/3354529013 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00032440 con 0x7fed1009e770 2024-09-15T03:58:45.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fecf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fecf7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:45.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fecf7fff640 10 monclient: dump: 2024-09-15T03:58:45.231 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.231 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.231 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.231 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.231 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.231 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.231 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.231 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.232 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.232 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fecf7fff640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fecf7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fecf7fff640 1 -- 172.21.15.110:0/3354529013 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed00032e60 con 0x7fed1009e770 2024-09-15T03:58:45.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fed1691d640 5 monclient: authenticate success, global_id 4250 2024-09-15T03:58:45.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fed1691d640 10 monclient: _renew_subs 2024-09-15T03:58:45.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fed1691d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fed1691d640 1 -- 172.21.15.110:0/3354529013 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed10097d50 con 0x7fed1009e770 2024-09-15T03:58:45.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fecf7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fecf7fff640 1 -- 172.21.15.110:0/3354529013 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00031ea0 con 0x7fed1009e770 2024-09-15T03:58:45.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fed0e7fc640 4 set_mon_vals no callback set 2024-09-15T03:58:45.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fed1691d640 10 monclient: _renew_subs 2024-09-15T03:58:45.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fed1691d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fed1691d640 1 -- 172.21.15.110:0/3354529013 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1009aa80 con 0x7fed1009e770 2024-09-15T03:58:45.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fecf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fecf7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:45.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.120+0000 7fecf7fff640 10 monclient: dump: 2024-09-15T03:58:45.233 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.233 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.233 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.233 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.233 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.233 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.233 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.233 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.234 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.234 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.121+0000 7f8146cf1640 1 --2- 172.21.15.110:0/2842859561 >> v2:172.21.15.110:6808/28367 conn(0x7f812403e1f0 0x7f8124040620 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.121+0000 7fed1691d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.121+0000 7fed1691d640 1 -- 172.21.15.110:0/3354529013 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fed100647e0 con 0x7fed1009e770 2024-09-15T03:58:45.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.122+0000 7fe84e0ba640 1 --2- 172.21.15.110:0/2390232608 >> v2:172.21.15.110:6800/28369 conn(0x7fe82c03e180 0x7fe82c0405b0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.122+0000 7f81367fc640 1 -- 172.21.15.110:0/2842859561 <== osd.0 v2:172.21.15.110:6808/28367 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f8124040cb0 con 0x7f812403e1f0 2024-09-15T03:58:45.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.122+0000 7fecf7fff640 1 -- 172.21.15.110:0/3354529013 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fed00039040 con 0x7fed1009e770 2024-09-15T03:58:45.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.122+0000 7fecf7fff640 1 --2- 172.21.15.110:0/3354529013 >> v2:172.21.15.177:6812/28201 conn(0x7fecfc038780 0x7fecfc03ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.122+0000 7fecf7fff640 1 -- 172.21.15.110:0/3354529013 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7fed00041030 con 0x7fed1009e770 2024-09-15T03:58:45.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.123+0000 7fecf7fff640 1 --2- 172.21.15.110:0/3354529013 >> v2:172.21.15.177:6800/28629 conn(0x7fecfc03e320 0x7fecfc040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.123+0000 7fecf7fff640 1 -- 172.21.15.110:0/3354529013 --> v2:172.21.15.177:6800/28629 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecfc040de0 con 0x7fecfc03e320 2024-09-15T03:58:45.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.123+0000 7fed0f7fe640 1 --2- 172.21.15.110:0/3354529013 >> v2:172.21.15.177:6812/28201 conn(0x7fecfc038780 0x7fecfc03ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.123+0000 7fed0f7fe640 10 monclient: get_auth_request con 0x7fecfc038780 auth_method 0 2024-09-15T03:58:45.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.123+0000 7fe83dffb640 1 -- 172.21.15.110:0/2390232608 <== osd.1 v2:172.21.15.110:6800/28369 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7fe82c040c40 con 0x7fe82c03e180 2024-09-15T03:58:45.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.123+0000 7fecf7fff640 1 -- 172.21.15.110:0/3354529013 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed0006d940 con 0x7fed1009e770 2024-09-15T03:58:45.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.123+0000 7fecf7fff640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-15T03:58:45.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.123+0000 7fed14eba640 1 --2- 172.21.15.110:0/3354529013 >> v2:172.21.15.177:6800/28629 conn(0x7fecfc03e320 0x7fecfc040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.123+0000 7fed14eba640 10 monclient: get_auth_request con 0x7fecfc03e320 auth_method 0 2024-09-15T03:58:45.238 INFO:teuthology.orchestra.run.smithi110.stdout:47244640259 2024-09-15T03:58:45.238 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:58:45.239 INFO:teuthology.orchestra.run.smithi110.stdout:47244640259 2024-09-15T03:58:45.239 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T03:58:45.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.126+0000 7fed0f7fe640 1 --2- 172.21.15.110:0/3354529013 >> v2:172.21.15.177:6812/28201 conn(0x7fecfc038780 0x7fecfc03ac30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fed04007fd0 tx=0x7fed04007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.129+0000 7fed14eba640 1 --2- 172.21.15.110:0/3354529013 >> v2:172.21.15.177:6800/28629 conn(0x7fecfc03e320 0x7fecfc040750 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.130+0000 7fecf7fff640 1 -- 172.21.15.110:0/3354529013 <== osd.4 v2:172.21.15.177:6800/28629 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7fecfc040de0 con 0x7fecfc03e320 2024-09-15T03:58:45.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.133+0000 7f78a6b9d640 1 -- 172.21.15.110:0/966291440 --> v2:172.21.15.110:6804/28368 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f78a006b920 con 0x7f788403e270 2024-09-15T03:58:45.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.133+0000 7f789cff9640 1 -- 172.21.15.110:0/966291440 <== osd.2 v2:172.21.15.110:6804/28368 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f78a006b920 con 0x7f788403e270 2024-09-15T03:58:45.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.133+0000 7f78a6b9d640 1 -- 172.21.15.110:0/966291440 >> v2:172.21.15.110:6804/28368 conn(0x7f788403e270 msgr2=0x7f78840406a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.133+0000 7f78a6b9d640 1 --2- 172.21.15.110:0/966291440 >> v2:172.21.15.110:6804/28368 conn(0x7f788403e270 0x7f78840406a0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.134+0000 7f78a6b9d640 1 -- 172.21.15.110:0/966291440 >> v2:172.21.15.177:6812/28201 conn(0x7f7884038750 msgr2=0x7f788403ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.134+0000 7f78a6b9d640 1 --2- 172.21.15.110:0/966291440 >> v2:172.21.15.177:6812/28201 conn(0x7f7884038750 0x7f788403ac00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f788c007fd0 tx=0x7f788c007a90 comp rx=0 tx=0).stop 2024-09-15T03:58:45.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.134+0000 7f78a6b9d640 10 monclient: shutdown 2024-09-15T03:58:45.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.134+0000 7f78a6b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.134+0000 7f78a6b9d640 1 -- 172.21.15.110:0/966291440 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78a009e770 msgr2=0x7f78a006a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.134+0000 7f78a6b9d640 1 --2- 172.21.15.110:0/966291440 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78a009e770 0x7f78a006a720 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7894017480 tx=0x7f7894008040 comp rx=0 tx=0).stop 2024-09-15T03:58:45.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.134+0000 7f78a513a640 1 -- 172.21.15.110:0/966291440 reap_dead start 2024-09-15T03:58:45.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.134+0000 7f78a6b9d640 1 -- 172.21.15.110:0/966291440 shutdown_connections 2024-09-15T03:58:45.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.134+0000 7f78a6b9d640 1 -- 172.21.15.110:0/966291440 >> 172.21.15.110:0/966291440 conn(0x7f78a006e860 msgr2=0x7f78a012a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.134+0000 7f78a6b9d640 1 -- 172.21.15.110:0/966291440 shutdown_connections 2024-09-15T03:58:45.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.134+0000 7f78a6b9d640 1 -- 172.21.15.110:0/966291440 wait complete. 2024-09-15T03:58:45.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.135+0000 7fe84fb1d640 1 -- 172.21.15.110:0/2390232608 --> v2:172.21.15.110:6800/28369 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe848117ad0 con 0x7fe82c03e180 2024-09-15T03:58:45.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.136+0000 7fe83dffb640 1 -- 172.21.15.110:0/2390232608 <== osd.1 v2:172.21.15.110:6800/28369 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe848117ad0 con 0x7fe82c03e180 2024-09-15T03:58:45.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7f8148754640 1 -- 172.21.15.110:0/2842859561 --> v2:172.21.15.110:6808/28367 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8140141290 con 0x7f812403e1f0 2024-09-15T03:58:45.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7fe84fb1d640 1 -- 172.21.15.110:0/2390232608 >> v2:172.21.15.110:6800/28369 conn(0x7fe82c03e180 msgr2=0x7fe82c0405b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7fe84fb1d640 1 --2- 172.21.15.110:0/2390232608 >> v2:172.21.15.110:6800/28369 conn(0x7fe82c03e180 0x7fe82c0405b0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7f81367fc640 1 -- 172.21.15.110:0/2842859561 <== osd.0 v2:172.21.15.110:6808/28367 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8140141290 con 0x7f812403e1f0 2024-09-15T03:58:45.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7fe84fb1d640 1 -- 172.21.15.110:0/2390232608 >> v2:172.21.15.177:6812/28201 conn(0x7fe82c038750 msgr2=0x7fe82c03ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7fe84fb1d640 1 --2- 172.21.15.110:0/2390232608 >> v2:172.21.15.177:6812/28201 conn(0x7fe82c038750 0x7fe82c03ac00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe838009cf0 tx=0x7fe838008040 comp rx=0 tx=0).stop 2024-09-15T03:58:45.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7fe84fb1d640 10 monclient: shutdown 2024-09-15T03:58:45.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7fe84fb1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7fe84fb1d640 1 -- 172.21.15.110:0/2390232608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8480a0290 msgr2=0x7fe848117050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7fe84fb1d640 1 --2- 172.21.15.110:0/2390232608 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe8480a0290 0x7fe848117050 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe84400b5e0 tx=0x7fe8440090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7fe84e0ba640 1 -- 172.21.15.110:0/2390232608 reap_dead start 2024-09-15T03:58:45.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7fe84fb1d640 1 -- 172.21.15.110:0/2390232608 shutdown_connections 2024-09-15T03:58:45.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7fe84fb1d640 1 -- 172.21.15.110:0/2390232608 >> 172.21.15.110:0/2390232608 conn(0x7fe84809d680 msgr2=0x7fe84809ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7fe84fb1d640 1 -- 172.21.15.110:0/2390232608 shutdown_connections 2024-09-15T03:58:45.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7fe84fb1d640 1 -- 172.21.15.110:0/2390232608 wait complete. 2024-09-15T03:58:45.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7f8148754640 1 -- 172.21.15.110:0/2842859561 >> v2:172.21.15.110:6808/28367 conn(0x7f812403e1f0 msgr2=0x7f8124040620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.138+0000 7f8148754640 1 --2- 172.21.15.110:0/2842859561 >> v2:172.21.15.110:6808/28367 conn(0x7f812403e1f0 0x7f8124040620 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.139+0000 7f8148754640 1 -- 172.21.15.110:0/2842859561 >> v2:172.21.15.177:6812/28201 conn(0x7f8124038750 msgr2=0x7f812403ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.139+0000 7f8148754640 1 --2- 172.21.15.110:0/2842859561 >> v2:172.21.15.177:6812/28201 conn(0x7f8124038750 0x7f812403ac00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8140068b30 tx=0x7f8130006c50 comp rx=0 tx=0).stop 2024-09-15T03:58:45.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.139+0000 7f8148754640 10 monclient: shutdown 2024-09-15T03:58:45.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.139+0000 7f8148754640 20 monclient: shutdown canceling and discarding version request 1 2024-09-15T03:58:45.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.139+0000 7f8148754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.139+0000 7f8148754640 1 -- 172.21.15.110:0/2842859561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140067130 msgr2=0x7f8140140d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.139+0000 7f8148754640 1 --2- 172.21.15.110:0/2842859561 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8140067130 0x7f8140140d50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f813c009210 tx=0x7f813c032000 comp rx=0 tx=0).stop 2024-09-15T03:58:45.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.139+0000 7f8146cf1640 1 -- 172.21.15.110:0/2842859561 reap_dead start 2024-09-15T03:58:45.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.139+0000 7f81367fc640 1 -- 172.21.15.110:0/2842859561 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f813c031ea0 con 0x7f8140067130 2024-09-15T03:58:45.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.139+0000 7f81367fc640 10 monclient: discarding stray monitor message mon_get_version_reply(handle=1 version=19) v2 2024-09-15T03:58:45.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.139+0000 7f8148754640 1 -- 172.21.15.110:0/2842859561 shutdown_connections 2024-09-15T03:58:45.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.139+0000 7f8148754640 1 -- 172.21.15.110:0/2842859561 >> 172.21.15.110:0/2842859561 conn(0x7f81400a2780 msgr2=0x7f8140134770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.139+0000 7f8148754640 1 -- 172.21.15.110:0/2842859561 shutdown_connections 2024-09-15T03:58:45.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.139+0000 7f8148754640 1 -- 172.21.15.110:0/2842859561 wait complete. 2024-09-15T03:58:45.249 INFO:teuthology.orchestra.run.smithi110.stdout:47244640259 2024-09-15T03:58:45.250 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:58:45.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.147+0000 7fed1691d640 1 -- 172.21.15.110:0/3354529013 --> v2:172.21.15.177:6800/28629 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fed10099cd0 con 0x7fecfc03e320 2024-09-15T03:58:45.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.147+0000 7fecf7fff640 1 -- 172.21.15.110:0/3354529013 <== osd.4 v2:172.21.15.177:6800/28629 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fed10099cd0 con 0x7fecfc03e320 2024-09-15T03:58:45.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.147+0000 7fed1691d640 1 -- 172.21.15.110:0/3354529013 >> v2:172.21.15.177:6800/28629 conn(0x7fecfc03e320 msgr2=0x7fecfc040750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.148+0000 7fed1691d640 1 --2- 172.21.15.110:0/3354529013 >> v2:172.21.15.177:6800/28629 conn(0x7fecfc03e320 0x7fecfc040750 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.148+0000 7fed1691d640 1 -- 172.21.15.110:0/3354529013 >> v2:172.21.15.177:6812/28201 conn(0x7fecfc038780 msgr2=0x7fecfc03ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.148+0000 7fed1691d640 1 --2- 172.21.15.110:0/3354529013 >> v2:172.21.15.177:6812/28201 conn(0x7fecfc038780 0x7fecfc03ac30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fed04007fd0 tx=0x7fed04007f00 comp rx=0 tx=0).stop 2024-09-15T03:58:45.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.148+0000 7fed1691d640 10 monclient: shutdown 2024-09-15T03:58:45.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.148+0000 7fed1691d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.148+0000 7fed1691d640 1 -- 172.21.15.110:0/3354529013 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed1009e770 msgr2=0x7fed1009a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.148+0000 7fed1691d640 1 --2- 172.21.15.110:0/3354529013 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fed1009e770 0x7fed1009a540 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fed00009880 tx=0x7fed00032000 comp rx=0 tx=0).stop 2024-09-15T03:58:45.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.148+0000 7fed14eba640 1 -- 172.21.15.110:0/3354529013 reap_dead start 2024-09-15T03:58:45.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.148+0000 7fed1691d640 1 -- 172.21.15.110:0/3354529013 shutdown_connections 2024-09-15T03:58:45.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.148+0000 7fed1691d640 1 -- 172.21.15.110:0/3354529013 >> 172.21.15.110:0/3354529013 conn(0x7fed1006e860 msgr2=0x7fed10138dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.148+0000 7fed1691d640 1 -- 172.21.15.110:0/3354529013 shutdown_connections 2024-09-15T03:58:45.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.148+0000 7fed1691d640 1 -- 172.21.15.110:0/3354529013 wait complete. 2024-09-15T03:58:45.253 INFO:teuthology.orchestra.run.smithi110.stdout:47244640259 2024-09-15T03:58:45.253 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:58:45.254 INFO:teuthology.orchestra.run.smithi110.stdout:47244640259 2024-09-15T03:58:45.254 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:58:45.255 INFO:teuthology.orchestra.run.smithi110.stdout:47244640259 2024-09-15T03:58:45.255 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:58:45.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.385+0000 7f1120c4f640 10 monclient: get_monmap_and_config 2024-09-15T03:58:45.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.385+0000 7f033fb1d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:45.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.385+0000 7f033fb1d640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.385+0000 7f033fb1d640 10 monclient: monmap: 2024-09-15T03:58:45.387 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.387 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.387 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:45.386472+0000 2024-09-15T03:58:45.387 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:45.386472+0000 2024-09-15T03:58:45.387 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.387 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.387 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:45.387 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:45.387 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:45.387 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:45.387 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:45.388 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.385+0000 7f1120c4f640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.385+0000 7f1120c4f640 10 monclient: monmap: 2024-09-15T03:58:45.388 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.388 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.388 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:45.386469+0000 2024-09-15T03:58:45.388 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:45.386469+0000 2024-09-15T03:58:45.388 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:45.389 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f033fb1d640 1 Processor -- start 2024-09-15T03:58:45.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 1 Processor -- start 2024-09-15T03:58:45.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 1 -- start start 2024-09-15T03:58:45.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 10 monclient: init 2024-09-15T03:58:45.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f033fb1d640 1 -- start start 2024-09-15T03:58:45.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f033fb1d640 10 monclient: init 2024-09-15T03:58:45.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-15T03:58:45.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f033fb1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f033fb1d640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-15T03:58:45.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f033fb1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03380a1cf0 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:45.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f111c0a1a10 0x7f111c131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 10 monclient(hunting): picked mon.noname-c con 0x7f111c0a1a10 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:45.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f111c09e770 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:45.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f111c1369e0 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:45.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111c09f2c0 con 0x7f111c1369e0 2024-09-15T03:58:45.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111c0a0850 con 0x7f111c09e770 2024-09-15T03:58:45.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f111c09c9a0 con 0x7f111c0a1a10 2024-09-15T03:58:45.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.386+0000 7f1120c4f640 10 monclient(hunting): authenticate will time out at 749.030964s 2024-09-15T03:58:45.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033fb1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0338131a80 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:45.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033fb1d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f033809ed70 0x7f033813a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111ad9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f111c1369e0 0x7f111c138dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33788/0 (socket says 172.21.15.110:33788) 2024-09-15T03:58:45.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033fb1d640 10 monclient(hunting): picked mon.noname-c con 0x7f033809ed70 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:45.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111ad9d640 1 -- 172.21.15.110:0/1227804585 learned_addr learned my addr 172.21.15.110:0/1227804585 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:45.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033fb1d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033809cb70 con 0x7f0338131a80 2024-09-15T03:58:45.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033d0b8640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f0338131a80 0x7f0338136940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33774/0 (socket says 172.21.15.110:33774) 2024-09-15T03:58:45.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033d0b8640 1 -- 172.21.15.110:0/757689498 learned_addr learned my addr 172.21.15.110:0/757689498 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:45.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 1 --2- 172.21.15.110:0/1227804585 >> v2:172.21.15.177:3300/0 conn(0x7f111c0a1a10 0x7f111c131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033fb1d640 1 -- 172.21.15.110:0/757689498 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03380a20c0 con 0x7f03380a1cf0 2024-09-15T03:58:45.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 10 monclient(hunting): get_auth_request con 0x7f111c0a1a10 auth_method 0 2024-09-15T03:58:45.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033e0ba640 1 --2- 172.21.15.110:0/757689498 >> v2:172.21.15.177:3300/0 conn(0x7f033809ed70 0x7f033813a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033fb1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033fb1d640 1 -- 172.21.15.110:0/757689498 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f033809c330 con 0x7f033809ed70 2024-09-15T03:58:45.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033fb1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033fb1d640 10 monclient(hunting): authenticate will time out at 749.031477s 2024-09-15T03:58:45.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033e0ba640 10 monclient(hunting): get_auth_request con 0x7f033809ed70 auth_method 0 2024-09-15T03:58:45.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033e0ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033e0ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f033e0ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 10 monclient(hunting): handle_auth_done global_id 4228 payload 274 2024-09-15T03:58:45.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 1 -- 172.21.15.110:0/1227804585 >> v1:172.21.15.177:6789/0 conn(0x7f111c09e770 legacy=0x7f111c09ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 1 -- 172.21.15.110:0/1227804585 >> v1:172.21.15.110:6789/0 conn(0x7f111c1369e0 legacy=0x7f111c138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 1 monclient: found mon.noname-c 2024-09-15T03:58:45.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:58:45.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.387+0000 7f111a59c640 1 -- 172.21.15.110:0/1227804585 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c09cc80 con 0x7f111c0a1a10 2024-09-15T03:58:45.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.389+0000 7f033e0ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.389+0000 7f033e0ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.389+0000 7f033e0ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.389+0000 7f033d8b9640 1 -- 172.21.15.110:0/757689498 >> v1:172.21.15.177:6789/0 conn(0x7f03380a1cf0 legacy=0x7f033809e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.389+0000 7f033d0b8640 1 -- 172.21.15.110:0/757689498 >> v1:172.21.15.110:6789/0 conn(0x7f0338131a80 legacy=0x7f0338136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.390+0000 7f111a59c640 1 -- 172.21.15.110:0/1227804585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f111c0a1a10 msgr2=0x7f111c131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.390+0000 7f111a59c640 1 --2- 172.21.15.110:0/1227804585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f111c0a1a10 0x7f111c131b90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f110c009850 tx=0x7f110c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=911231b3beb3da62 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.389+0000 7f033e0ba640 10 monclient(hunting): handle_auth_done global_id 4231 payload 274 2024-09-15T03:58:45.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.389+0000 7f033e0ba640 1 -- 172.21.15.110:0/757689498 >> v1:172.21.15.177:6789/0 conn(0x7f03380a1cf0 legacy=0x7f033809e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.389+0000 7f033e0ba640 1 -- 172.21.15.110:0/757689498 >> v1:172.21.15.110:6789/0 conn(0x7f0338131a80 legacy=0x7f0338136940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.389+0000 7f033e0ba640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.389+0000 7f033e0ba640 1 monclient: found mon.noname-c 2024-09-15T03:58:45.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.389+0000 7f033e0ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.389+0000 7f033e0ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:58:45.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.389+0000 7f033e0ba640 1 -- 172.21.15.110:0/757689498 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0338131fb0 con 0x7f033809ed70 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.390+0000 7f111959a640 1 -- 172.21.15.110:0/1227804585 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c032440 con 0x7f111c0a1a10 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.390+0000 7f111959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.390+0000 7f111959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f111959a640 10 monclient: dump: 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f111959a640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f111959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f111959a640 1 -- 172.21.15.110:0/1227804585 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110c0325a0 con 0x7f111c0a1a10 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f111959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f111959a640 1 -- 172.21.15.110:0/1227804585 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c032940 con 0x7f111c0a1a10 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f111959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f111959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f111959a640 10 monclient: dump: 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.403 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f1120c4f640 5 monclient: authenticate success, global_id 4228 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f1120c4f640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f1120c4f640 4 set_mon_vals no callback set 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f1120c4f640 10 monclient: shutdown 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f1120c4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f1120c4f640 1 -- 172.21.15.110:0/1227804585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f111c0a1a10 msgr2=0x7f111c131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.391+0000 7f1120c4f640 1 --2- 172.21.15.110:0/1227804585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f111c0a1a10 0x7f111c131b90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f110c009850 tx=0x7f110c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.392+0000 7f1120c4f640 1 -- 172.21.15.110:0/1227804585 shutdown_connections 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.392+0000 7f1120c4f640 1 --2- 172.21.15.110:0/1227804585 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f111c0a1a10 0x7f111c131b90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.392+0000 7f1120c4f640 1 -- 172.21.15.110:0/1227804585 >> 172.21.15.110:0/1227804585 conn(0x7f111c06e860 msgr2=0x7f111c06ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.392+0000 7f1120c4f640 1 -- 172.21.15.110:0/1227804585 shutdown_connections 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.392+0000 7f1120c4f640 1 -- 172.21.15.110:0/1227804585 wait complete. 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.392+0000 7f1120c4f640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f1120c4f640 10 monclient: monmap: 2024-09-15T03:58:45.405 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.405 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:45.405 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:45.405 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:45.405 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.405 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.405 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:45.405 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:45.405 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:45.405 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033e0ba640 1 -- 172.21.15.110:0/757689498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033809ed70 msgr2=0x7f033813a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033e0ba640 1 --2- 172.21.15.110:0/757689498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033809ed70 0x7f033813a070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f03381321a0 tx=0x7f0330009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e990d2c49fc4776 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f1120c4f640 1 Processor -- start 2024-09-15T03:58:45.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033c8b7640 1 -- 172.21.15.110:0/757689498 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0330007840 con 0x7f033809ed70 2024-09-15T03:58:45.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f1120c4f640 1 -- start start 2024-09-15T03:58:45.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f1120c4f640 10 monclient: init 2024-09-15T03:58:45.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033c8b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033c8b7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:45.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033c8b7640 10 monclient: dump: 2024-09-15T03:58:45.408 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.408 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.408 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.408 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.408 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.408 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.408 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.409 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.409 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.409 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033c8b7640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033c8b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033c8b7640 1 -- 172.21.15.110:0/757689498 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033002d050 con 0x7f033809ed70 2024-09-15T03:58:45.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033c8b7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033c8b7640 1 -- 172.21.15.110:0/757689498 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0330007bd0 con 0x7f033809ed70 2024-09-15T03:58:45.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033c8b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033c8b7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:45.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033c8b7640 10 monclient: dump: 2024-09-15T03:58:45.409 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.409 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.410 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.410 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.410 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.410 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.410 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.410 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.410 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.410 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033fb1d640 5 monclient: authenticate success, global_id 4231 2024-09-15T03:58:45.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033fb1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:45.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.393+0000 7f033fb1d640 4 set_mon_vals no callback set 2024-09-15T03:58:45.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:58:45.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f111c09e770 0x7f111c133a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 10 monclient(hunting): picked mon.noname-b con 0x7f111c09e770 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:45.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c0a1a10 0x7f111c133fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 10 monclient(hunting): picked mon.noname-a con 0x7f111c0a1a10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:45.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c1369e0 0x7f111c1344e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 10 monclient(hunting): picked mon.noname-c con 0x7f111c1369e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c12d5a0 con 0x7f111c0a1a10 2024-09-15T03:58:45.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c12d2c0 con 0x7f111c09e770 2024-09-15T03:58:45.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111c12d880 con 0x7f111c1369e0 2024-09-15T03:58:45.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f033fb1d640 10 monclient: shutdown 2024-09-15T03:58:45.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f033fb1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f033fb1d640 1 -- 172.21.15.110:0/757689498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033809ed70 msgr2=0x7f033813a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f033fb1d640 1 --2- 172.21.15.110:0/757689498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033809ed70 0x7f033813a070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f03381321a0 tx=0x7f0330009780 comp rx=0 tx=0).stop 2024-09-15T03:58:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1120c4f640 10 monclient(hunting): authenticate will time out at 749.038315s 2024-09-15T03:58:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111ad9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c1369e0 0x7f111c1344e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111ad9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c1369e0 0x7f111c1344e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55008/0 (socket says 172.21.15.110:55008) 2024-09-15T03:58:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111ad9d640 1 -- 172.21.15.110:0/3512942837 learned_addr learned my addr 172.21.15.110:0/3512942837 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111ad9d640 10 monclient(hunting): get_auth_request con 0x7f111c1369e0 auth_method 0 2024-09-15T03:58:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111a59c640 1 --2- 172.21.15.110:0/3512942837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f111c09e770 0x7f111c133a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111a59c640 10 monclient(hunting): get_auth_request con 0x7f111c09e770 auth_method 0 2024-09-15T03:58:45.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f033fb1d640 1 -- 172.21.15.110:0/757689498 shutdown_connections 2024-09-15T03:58:45.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f033fb1d640 1 --2- 172.21.15.110:0/757689498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033809ed70 0x7f033813a070 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f033fb1d640 1 -- 172.21.15.110:0/757689498 >> 172.21.15.110:0/757689498 conn(0x7f0338062f70 msgr2=0x7f0338063340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f111ad9d640 10 monclient(hunting): handle_auth_done global_id 4253 payload 824 2024-09-15T03:58:45.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.394+0000 7f1119d9b640 1 --2- 172.21.15.110:0/3512942837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c0a1a10 0x7f111c133fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f111ad9d640 1 -- 172.21.15.110:0/3512942837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f111c09e770 msgr2=0x7f111c133a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f111ad9d640 1 --2- 172.21.15.110:0/3512942837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f111c09e770 0x7f111c133a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f111ad9d640 1 -- 172.21.15.110:0/3512942837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c0a1a10 msgr2=0x7f111c133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f111ad9d640 1 --2- 172.21.15.110:0/3512942837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c0a1a10 0x7f111c133fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f111ad9d640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f111ad9d640 1 monclient: found mon.noname-c 2024-09-15T03:58:45.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f111ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f111ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f111ad9d640 1 -- 172.21.15.110:0/3512942837 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110c009390 con 0x7f111c1369e0 2024-09-15T03:58:45.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f033fb1d640 1 -- 172.21.15.110:0/757689498 shutdown_connections 2024-09-15T03:58:45.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f033fb1d640 1 -- 172.21.15.110:0/757689498 wait complete. 2024-09-15T03:58:45.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f1119d9b640 10 monclient: get_auth_request con 0x7f111c0a1a10 auth_method 0 2024-09-15T03:58:45.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f1119d9b640 1 --2- 172.21.15.110:0/3512942837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c0a1a10 0x7f111c133fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:58:45.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f033fb1d640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.395+0000 7f033fb1d640 10 monclient: monmap: 2024-09-15T03:58:45.420 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.420 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:45.420 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:45.420 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:45.420 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.420 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.420 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:45.420 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:45.420 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:45.420 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 1 Processor -- start 2024-09-15T03:58:45.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 1 -- start start 2024-09-15T03:58:45.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 10 monclient: init 2024-09-15T03:58:45.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:58:45.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033809ed70 0x7f033806db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 10 monclient(hunting): picked mon.noname-b con 0x7f033809ed70 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:45.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03380a1cf0 0x7f033806a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 10 monclient(hunting): picked mon.noname-a con 0x7f03380a1cf0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:45.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0338131a80 0x7f033806a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 10 monclient(hunting): picked mon.noname-c con 0x7f0338131a80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033806cc20 con 0x7f03380a1cf0 2024-09-15T03:58:45.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03380a59d0 con 0x7f033809ed70 2024-09-15T03:58:45.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033806d600 con 0x7f0338131a80 2024-09-15T03:58:45.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033fb1d640 10 monclient(hunting): authenticate will time out at 749.040695s 2024-09-15T03:58:45.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d0b8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03380a1cf0 0x7f033806a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d8b9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033809ed70 0x7f033806db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033e0ba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0338131a80 0x7f033806a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d0b8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03380a1cf0 0x7f033806a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56492/0 (socket says 172.21.15.110:56492) 2024-09-15T03:58:45.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d0b8640 1 -- 172.21.15.110:0/444383082 learned_addr learned my addr 172.21.15.110:0/444383082 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d0b8640 10 monclient(hunting): get_auth_request con 0x7f03380a1cf0 auth_method 0 2024-09-15T03:58:45.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d0b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d0b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d0b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033e0ba640 10 monclient(hunting): get_auth_request con 0x7f0338131a80 auth_method 0 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033e0ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033e0ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033e0ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d8b9640 10 monclient(hunting): get_auth_request con 0x7f033809ed70 auth_method 0 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d8b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d8b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d8b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d0b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d0b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033d0b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033e0ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033e0ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.396+0000 7f033e0ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f033d8b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f033d8b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f033d8b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f033e0ba640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2024-09-15T03:58:45.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f033e0ba640 1 -- 172.21.15.110:0/444383082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033809ed70 msgr2=0x7f033806db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f033e0ba640 1 --2- 172.21.15.110:0/444383082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033809ed70 0x7f033806db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f033e0ba640 1 -- 172.21.15.110:0/444383082 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03380a1cf0 msgr2=0x7f033806a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f033e0ba640 1 --2- 172.21.15.110:0/444383082 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03380a1cf0 0x7f033806a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f033e0ba640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f033e0ba640 1 monclient: found mon.noname-c 2024-09-15T03:58:45.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f033e0ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f033e0ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f033e0ba640 1 -- 172.21.15.110:0/444383082 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0330009a20 con 0x7f0338131a80 2024-09-15T03:58:45.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f111ad9d640 1 -- 172.21.15.110:0/3512942837 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c1369e0 msgr2=0x7f111c1344e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f111ad9d640 1 --2- 172.21.15.110:0/3512942837 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c1369e0 0x7f111c1344e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f111400ccc0 tx=0x7f1114005370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f110a7fc640 1 -- 172.21.15.110:0/3512942837 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111400ce80 con 0x7f111c1369e0 2024-09-15T03:58:45.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f110a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f110a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:45.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f110a7fc640 10 monclient: dump: 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f110a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f110a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f110a7fc640 1 -- 172.21.15.110:0/3512942837 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111400aa60 con 0x7f111c1369e0 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f110a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.397+0000 7f1118d99640 4 set_mon_vals no callback set 2024-09-15T03:58:45.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.398+0000 7f1120c4f640 5 monclient: authenticate success, global_id 4253 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.398+0000 7f110a7fc640 1 -- 172.21.15.110:0/3512942837 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111400ae20 con 0x7f111c1369e0 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.398+0000 7f1120c4f640 10 monclient: _renew_subs 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.398+0000 7f1120c4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.398+0000 7f1120c4f640 1 -- 172.21.15.110:0/3512942837 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111c12db60 con 0x7f111c1369e0 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.398+0000 7f110a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.398+0000 7f110a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.398+0000 7f110a7fc640 10 monclient: dump: 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.427 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.428 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.428 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.398+0000 7f1120c4f640 10 monclient: _renew_subs 2024-09-15T03:58:45.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.398+0000 7f1120c4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.398+0000 7f1120c4f640 1 -- 172.21.15.110:0/3512942837 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111c135290 con 0x7f111c1369e0 2024-09-15T03:58:45.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.399+0000 7f110a7fc640 1 -- 172.21.15.110:0/3512942837 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f1114007300 con 0x7f111c1369e0 2024-09-15T03:58:45.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.399+0000 7f033e0ba640 1 -- 172.21.15.110:0/444383082 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0338131a80 msgr2=0x7f033806a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.399+0000 7f033e0ba640 1 --2- 172.21.15.110:0/444383082 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0338131a80 0x7f033806a990 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0330034000 tx=0x7f0330037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.399+0000 7f110a7fc640 1 --2- 172.21.15.110:0/3512942837 >> v2:172.21.15.177:6812/28201 conn(0x7f10fc038750 0x7f10fc03ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.399+0000 7f1120c4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.399+0000 7f110a7fc640 1 -- 172.21.15.110:0/3512942837 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f111404b710 con 0x7f111c1369e0 2024-09-15T03:58:45.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.399+0000 7f1120c4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.399+0000 7f1120c4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.399+0000 7f1120c4f640 1 -- 172.21.15.110:0/3512942837 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111c0647e0 con 0x7f111c1369e0 2024-09-15T03:58:45.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.399+0000 7f111a59c640 1 --2- 172.21.15.110:0/3512942837 >> v2:172.21.15.177:6812/28201 conn(0x7f10fc038750 0x7f10fc03ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.430 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f111a59c640 10 monclient: get_auth_request con 0x7f10fc038750 auth_method 0 2024-09-15T03:58:45.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f032dffb640 1 -- 172.21.15.110:0/444383082 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0330031e20 con 0x7f0338131a80 2024-09-15T03:58:45.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f032dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f032dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:45.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f032dffb640 10 monclient: dump: 2024-09-15T03:58:45.431 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.431 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.431 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.432 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.432 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.432 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.432 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.432 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.432 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.432 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f032dffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f032dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f032dffb640 1 -- 172.21.15.110:0/444383082 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f033003c420 con 0x7f0338131a80 2024-09-15T03:58:45.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f033fb1d640 5 monclient: authenticate success, global_id 4256 2024-09-15T03:58:45.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f033fb1d640 10 monclient: _renew_subs 2024-09-15T03:58:45.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f033fb1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.432 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f033fb1d640 1 -- 172.21.15.110:0/444383082 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033806c940 con 0x7f0338131a80 2024-09-15T03:58:45.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f032dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f032dffb640 1 -- 172.21.15.110:0/444383082 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033003c7e0 con 0x7f0338131a80 2024-09-15T03:58:45.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f032dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f032dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:45.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f032dffb640 10 monclient: dump: 2024-09-15T03:58:45.433 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.433 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.433 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.433 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.433 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.433 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.433 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.433 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.433 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.434 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f033fb1d640 10 monclient: _renew_subs 2024-09-15T03:58:45.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f033fb1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f033fb1d640 1 -- 172.21.15.110:0/444383082 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033809a5c0 con 0x7f0338131a80 2024-09-15T03:58:45.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.400+0000 7f032ffff640 4 set_mon_vals no callback set 2024-09-15T03:58:45.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.401+0000 7f03237fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.401+0000 7f03237fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.402+0000 7f03237fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.402+0000 7f03237fe640 1 -- 172.21.15.110:0/444383082 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03380a1250 con 0x7f0338131a80 2024-09-15T03:58:45.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.402+0000 7f111a59c640 1 --2- 172.21.15.110:0/3512942837 >> v2:172.21.15.177:6812/28201 conn(0x7f10fc038750 0x7f10fc03ac00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f111c0a0c30 tx=0x7f110c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.408+0000 7f110a7fc640 1 -- 172.21.15.110:0/3512942837 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f111401b400 con 0x7f111c1369e0 2024-09-15T03:58:45.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.408+0000 7f110a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.408+0000 7f110a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:45.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.409+0000 7f032dffb640 1 -- 172.21.15.110:0/444383082 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f033003cb90 con 0x7f0338131a80 2024-09-15T03:58:45.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.409+0000 7f032dffb640 1 --2- 172.21.15.110:0/444383082 >> v2:172.21.15.177:6812/28201 conn(0x7f031c038750 0x7f031c03ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.410+0000 7f032dffb640 1 -- 172.21.15.110:0/444383082 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f033006cf20 con 0x7f0338131a80 2024-09-15T03:58:45.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.410+0000 7f033d8b9640 1 --2- 172.21.15.110:0/444383082 >> v2:172.21.15.177:6812/28201 conn(0x7f031c038750 0x7f031c03ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.410+0000 7f033d8b9640 10 monclient: get_auth_request con 0x7f031c038750 auth_method 0 2024-09-15T03:58:45.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.413+0000 7f033d8b9640 1 --2- 172.21.15.110:0/444383082 >> v2:172.21.15.177:6812/28201 conn(0x7f031c038750 0x7f031c03ac00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0334007fd0 tx=0x7f0334009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.415+0000 7f032dffb640 1 -- 172.21.15.110:0/444383082 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f033003ce40 con 0x7f0338131a80 2024-09-15T03:58:45.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.415+0000 7f032dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.415+0000 7f032dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:45.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.421+0000 7fd1cc846640 10 monclient: get_monmap_and_config 2024-09-15T03:58:45.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.421+0000 7fd1cc846640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.439 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.421+0000 7fd1cc846640 10 monclient: monmap: 2024-09-15T03:58:45.439 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.439 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.439 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:45.422022+0000 2024-09-15T03:58:45.439 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:45.422022+0000 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.421+0000 7fd1cc846640 1 Processor -- start 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.421+0000 7fd1cc846640 1 -- start start 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.421+0000 7fd1cc846640 10 monclient: init 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.422+0000 7fd1cc846640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.422+0000 7fd1cc846640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.422+0000 7fd1cc846640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1c80a1a10 0x7fd1c8131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.422+0000 7fd1cc846640 10 monclient(hunting): picked mon.noname-b con 0x7fd1c80a1a10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.422+0000 7fd1cc846640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1c809e770 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:45.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.422+0000 7fd1c659c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1c80a1a10 0x7fd1c8131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.422+0000 7fd1c659c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1c80a1a10 0x7fd1c8131b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55018/0 (socket says 172.21.15.110:55018) 2024-09-15T03:58:45.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.422+0000 7fd1c659c640 1 -- 172.21.15.110:0/1390679094 learned_addr learned my addr 172.21.15.110:0/1390679094 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.422+0000 7fd1cc846640 1 --2- 172.21.15.110:0/1390679094 >> v2:172.21.15.177:3300/0 conn(0x7fd1c81369e0 0x7fd1c8138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.422+0000 7fd1cc846640 10 monclient(hunting): picked mon.noname-c con 0x7fd1c81369e0 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:45.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.422+0000 7fd1cc846640 1 -- 172.21.15.110:0/1390679094 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c809f290 con 0x7fd1c809e770 2024-09-15T03:58:45.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.422+0000 7fd1cc846640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1cc846640 1 -- 172.21.15.110:0/1390679094 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1c80a0850 con 0x7fd1c81369e0 2024-09-15T03:58:45.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1cc846640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1cc846640 1 -- 172.21.15.110:0/1390679094 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c809c9a0 con 0x7fd1c80a1a10 2024-09-15T03:58:45.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7f3051f54640 10 monclient: get_monmap_and_config 2024-09-15T03:58:45.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7f3051f54640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1cc846640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7f3051f54640 10 monclient: monmap: 2024-09-15T03:58:45.442 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.442 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.442 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:45.424147+0000 2024-09-15T03:58:45.443 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:45.424147+0000 2024-09-15T03:58:45.443 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.443 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.443 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:45.443 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:45.443 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:45.443 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:45.443 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:45.443 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1cc846640 10 monclient(hunting): authenticate will time out at 749.067050s 2024-09-15T03:58:45.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 10 monclient(hunting): get_auth_request con 0x7fd1c80a1a10 auth_method 0 2024-09-15T03:58:45.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c6d9d640 1 --2- 172.21.15.110:0/1390679094 >> v2:172.21.15.177:3300/0 conn(0x7fd1c81369e0 0x7fd1c8138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c6d9d640 10 monclient(hunting): get_auth_request con 0x7fd1c81369e0 auth_method 0 2024-09-15T03:58:45.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 10 monclient(hunting): handle_auth_done global_id 4259 payload 274 2024-09-15T03:58:45.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 1 -- 172.21.15.110:0/1390679094 >> v2:172.21.15.177:3300/0 conn(0x7fd1c81369e0 msgr2=0x7fd1c8138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 1 --2- 172.21.15.110:0/1390679094 >> v2:172.21.15.177:3300/0 conn(0x7fd1c81369e0 0x7fd1c8138dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 1 -- 172.21.15.110:0/1390679094 >> v1:172.21.15.110:6789/0 conn(0x7fd1c809e770 legacy=0x7fd1c809eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 1 monclient: found mon.noname-b 2024-09-15T03:58:45.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.446 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.423+0000 7fd1c659c640 1 -- 172.21.15.110:0/1390679094 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c809cc80 con 0x7fd1c80a1a10 2024-09-15T03:58:45.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.424+0000 7f3051f54640 1 Processor -- start 2024-09-15T03:58:45.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.424+0000 7f3051f54640 1 -- start start 2024-09-15T03:58:45.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.424+0000 7f3051f54640 10 monclient: init 2024-09-15T03:58:45.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.424+0000 7f3051f54640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.424+0000 7f3051f54640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T03:58:45.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304b7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f304c0a1a10 0x7f304c09e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33814/0 (socket says 172.21.15.110:33814) 2024-09-15T03:58:45.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304b7fe640 1 -- 172.21.15.110:0/1106872379 learned_addr learned my addr 172.21.15.110:0/1106872379 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:45.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3051f54640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f304c0a1a10 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:45.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3051f54640 1 --2- 172.21.15.110:0/1106872379 >> v2:172.21.15.177:3300/0 conn(0x7f304c09ee80 0x7f304c131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3051f54640 10 monclient(hunting): picked mon.noname-c con 0x7f304c09ee80 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:45.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3051f54640 1 --2- 172.21.15.110:0/1106872379 >> v2:172.21.15.110:3300/0 conn(0x7f304c1369e0 0x7f304c138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3051f54640 10 monclient(hunting): picked mon.noname-a con 0x7f304c1369e0 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:45.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3051f54640 1 -- 172.21.15.110:0/1106872379 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c09f2c0 con 0x7f304c0a1a10 2024-09-15T03:58:45.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3051f54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3051f54640 1 -- 172.21.15.110:0/1106872379 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304c0a0850 con 0x7f304c1369e0 2024-09-15T03:58:45.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3051f54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3051f54640 1 -- 172.21.15.110:0/1106872379 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304c09c9a0 con 0x7f304c09ee80 2024-09-15T03:58:45.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3051f54640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3051f54640 10 monclient(hunting): authenticate will time out at 749.069198s 2024-09-15T03:58:45.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 1 --2- 172.21.15.110:0/1106872379 >> v2:172.21.15.177:3300/0 conn(0x7f304c09ee80 0x7f304c131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 10 monclient(hunting): get_auth_request con 0x7f304c09ee80 auth_method 0 2024-09-15T03:58:45.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3d90c3f640 10 monclient: get_monmap_and_config 2024-09-15T03:58:45.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3d90c3f640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f3d90c3f640 10 monclient: monmap: 2024-09-15T03:58:45.451 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.451 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.451 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:45.426561+0000 2024-09-15T03:58:45.451 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:45.426561+0000 2024-09-15T03:58:45.451 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.451 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.451 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:45.451 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:45.451 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:45.451 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:45.451 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:45.451 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 10 monclient(hunting): handle_auth_done global_id 4246 payload 274 2024-09-15T03:58:45.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f5d07a6f640 10 monclient: get_monmap_and_config 2024-09-15T03:58:45.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f5d07a6f640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f5d07a6f640 10 monclient: monmap: 2024-09-15T03:58:45.452 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.452 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.452 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:45.426676+0000 2024-09-15T03:58:45.452 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:45.426676+0000 2024-09-15T03:58:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:45.453 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:45.453 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 1 -- 172.21.15.110:0/1106872379 >> v2:172.21.15.110:3300/0 conn(0x7f304c1369e0 msgr2=0x7f304c138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 1 --2- 172.21.15.110:0/1106872379 >> v2:172.21.15.110:3300/0 conn(0x7f304c1369e0 0x7f304c138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 1 -- 172.21.15.110:0/1106872379 >> v1:172.21.15.110:6789/0 conn(0x7f304c0a1a10 legacy=0x7f304c09e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 1 monclient: found mon.noname-c 2024-09-15T03:58:45.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:58:45.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.425+0000 7f304affd640 1 -- 172.21.15.110:0/1106872379 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c09cc80 con 0x7f304c09ee80 2024-09-15T03:58:45.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7f3d90c3f640 1 Processor -- start 2024-09-15T03:58:45.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7f3d90c3f640 1 -- start start 2024-09-15T03:58:45.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7f3d90c3f640 10 monclient: init 2024-09-15T03:58:45.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7f5d07a6f640 1 Processor -- start 2024-09-15T03:58:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1c659c640 1 -- 172.21.15.110:0/1390679094 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1c80a1a10 msgr2=0x7fd1c8131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1c659c640 1 --2- 172.21.15.110:0/1390679094 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1c80a1a10 0x7fd1c8131b90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009910 tx=0x7fd1b80091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68e26bbaae95473d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7f5d07a6f640 1 -- start start 2024-09-15T03:58:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1c5d9b640 1 -- 172.21.15.110:0/1390679094 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8032440 con 0x7fd1c80a1a10 2024-09-15T03:58:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1c5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1c5d9b640 10 monclient: dump: 2024-09-15T03:58:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.457 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1c5d9b640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1cc846640 5 monclient: authenticate success, global_id 4259 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1cc846640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1c5d9b640 1 -- 172.21.15.110:0/1390679094 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b80325a0 con 0x7fd1c80a1a10 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1c5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1c5d9b640 1 -- 172.21.15.110:0/1390679094 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8032940 con 0x7fd1c80a1a10 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1cc846640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:45.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1cc846640 4 set_mon_vals no callback set 2024-09-15T03:58:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1c5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7fd1c5d9b640 10 monclient: dump: 2024-09-15T03:58:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.459 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.459 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.426+0000 7f5d07a6f640 10 monclient: init 2024-09-15T03:58:45.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d90c3f640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d90c3f640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-15T03:58:45.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d90c3f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d8c0a1a10 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:45.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d90c3f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3d8c09ee80 0x7f3d8c131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d90c3f640 10 monclient(hunting): picked mon.noname-a con 0x7f3d8c09ee80 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:45.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d90c3f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d8c1369e0 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:45.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d90c3f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d8c09f2c0 con 0x7f3d8c0a1a10 2024-09-15T03:58:45.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d90c3f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d8c0a0850 con 0x7f3d8c1369e0 2024-09-15T03:58:45.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d90c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d90c3f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d8c09c9a0 con 0x7f3d8c09ee80 2024-09-15T03:58:45.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d90c3f640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d90c3f640 10 monclient(hunting): authenticate will time out at 749.071313s 2024-09-15T03:58:45.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d07a6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d07a6f640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-15T03:58:45.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d07a6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d001315a0 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:45.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d07a6f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5d0006e610 0x7f5d0006e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d07a6f640 10 monclient(hunting): picked mon.noname-c con 0x7f5d0006e610 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:45.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d07a6f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d001368d0 0x7f5d00138cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d07a6f640 10 monclient(hunting): picked mon.noname-b con 0x7f5d001368d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d07a6f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d000699c0 con 0x7f5d001315a0 2024-09-15T03:58:45.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d07a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d07a6f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d000696e0 con 0x7f5d0006e610 2024-09-15T03:58:45.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d07a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d07a6f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0006ef20 con 0x7f5d001368d0 2024-09-15T03:58:45.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d07a6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d07a6f640 10 monclient(hunting): authenticate will time out at 749.071428s 2024-09-15T03:58:45.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3d8c09ee80 0x7f3d8c131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3d8c09ee80 0x7f3d8c131b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56516/0 (socket says 172.21.15.110:56516) 2024-09-15T03:58:45.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 1 -- 172.21.15.110:0/922765203 learned_addr learned my addr 172.21.15.110:0/922765203 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 10 monclient(hunting): get_auth_request con 0x7f3d8c09ee80 auth_method 0 2024-09-15T03:58:45.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7fd1cc846640 10 monclient: shutdown 2024-09-15T03:58:45.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.464 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7fd1cc846640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0500a640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5d0006e610 0x7f5d0006e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0500a640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5d0006e610 0x7f5d0006e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.110:59820/0 (socket says 172.21.15.110:59820) 2024-09-15T03:58:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0500a640 1 -- 172.21.15.110:0/2595153138 learned_addr learned my addr 172.21.15.110:0/2595153138 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0500a640 10 monclient(hunting): get_auth_request con 0x7f5d0006e610 auth_method 0 2024-09-15T03:58:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0500a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0500a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0500a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7fd1cc846640 1 -- 172.21.15.110:0/1390679094 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1c80a1a10 msgr2=0x7fd1c8131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7fd1cc846640 1 --2- 172.21.15.110:0/1390679094 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1c80a1a10 0x7fd1c8131b90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009910 tx=0x7fd1b80091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7fd1cc846640 1 -- 172.21.15.110:0/1390679094 shutdown_connections 2024-09-15T03:58:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0500a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0500a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0500a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7fd1cc846640 1 --2- 172.21.15.110:0/1390679094 >> v2:172.21.15.177:3300/0 conn(0x7fd1c81369e0 0x7fd1c8138dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7fd1cc846640 1 --2- 172.21.15.110:0/1390679094 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1c80a1a10 0x7fd1c8131b90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.466 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7fd1cc846640 1 -- 172.21.15.110:0/1390679094 >> 172.21.15.110:0/1390679094 conn(0x7fd1c806e860 msgr2=0x7fd1c806ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 10 monclient(hunting): handle_auth_done global_id 4245 payload 274 2024-09-15T03:58:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 1 -- 172.21.15.110:0/922765203 >> v1:172.21.15.177:6789/0 conn(0x7f3d8c1369e0 legacy=0x7f3d8c138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 1 -- 172.21.15.110:0/922765203 >> v1:172.21.15.110:6789/0 conn(0x7f3d8c0a1a10 legacy=0x7f3d8c09e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 1 monclient: found mon.noname-a 2024-09-15T03:58:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f3d89d9b640 1 -- 172.21.15.110:0/922765203 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c09cc80 con 0x7f3d8c09ee80 2024-09-15T03:58:45.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7fd1cc846640 1 -- 172.21.15.110:0/1390679094 shutdown_connections 2024-09-15T03:58:45.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7fd1cc846640 1 -- 172.21.15.110:0/1390679094 wait complete. 2024-09-15T03:58:45.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0600c640 1 --2- 172.21.15.110:0/2595153138 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d001368d0 0x7f5d00138cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0600c640 10 monclient(hunting): get_auth_request con 0x7f5d001368d0 auth_method 0 2024-09-15T03:58:45.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0600c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0600c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.427+0000 7f5d0600c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7f5d0600c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7f5d0600c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7f5d0600c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7f5d0500a640 10 monclient(hunting): handle_auth_done global_id 4249 payload 274 2024-09-15T03:58:45.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7f5d0500a640 1 -- 172.21.15.110:0/2595153138 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d001368d0 msgr2=0x7f5d00138cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7f5d0500a640 1 --2- 172.21.15.110:0/2595153138 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d001368d0 0x7f5d00138cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7f5d0500a640 1 -- 172.21.15.110:0/2595153138 >> v1:172.21.15.177:6789/0 conn(0x7f5d001315a0 legacy=0x7f5d00131970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:45.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7f5d0500a640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7f5d0500a640 1 monclient: found mon.noname-c 2024-09-15T03:58:45.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7f5d0500a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7f5d0500a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:58:45.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7f5d0500a640 1 -- 172.21.15.110:0/2595153138 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0009f350 con 0x7f5d0006e610 2024-09-15T03:58:45.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7fd1cc846640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7fd1cc846640 10 monclient: monmap: 2024-09-15T03:58:45.470 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.470 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:45.470 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:45.470 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:45.471 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7f304affd640 1 -- 172.21.15.110:0/1106872379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c09ee80 msgr2=0x7f304c131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.428+0000 7f304affd640 1 --2- 172.21.15.110:0/1106872379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c09ee80 0x7f304c131b90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f304c0a2790 tx=0x7f30400090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42e7580d38ef65fb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 1 Processor -- start 2024-09-15T03:58:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f304a7fc640 1 -- 172.21.15.110:0/1106872379 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040032440 con 0x7f304c09ee80 2024-09-15T03:58:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f304a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 1 -- start start 2024-09-15T03:58:45.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 10 monclient: init 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f304a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f304a7fc640 10 monclient: dump: 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f304a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f304a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f304a7fc640 1 -- 172.21.15.110:0/1106872379 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30400325a0 con 0x7f304c09ee80 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f304a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f304a7fc640 1 -- 172.21.15.110:0/1106872379 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040032940 con 0x7f304c09ee80 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f304a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f304a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f304a7fc640 10 monclient: dump: 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f3051f54640 5 monclient: authenticate success, global_id 4246 2024-09-15T03:58:45.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f3051f54640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:45.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f3051f54640 4 set_mon_vals no callback set 2024-09-15T03:58:45.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:58:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1c809e770 0x7fd1c8131060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 10 monclient(hunting): picked mon.noname-a con 0x7fd1c809e770 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1c80a1a10 0x7fd1c8096f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 10 monclient(hunting): picked mon.noname-b con 0x7fd1c80a1a10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1c81369e0 0x7fd1c80974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1c809e770 0x7fd1c8131060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1c809e770 0x7fd1c8131060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56532/0 (socket says 172.21.15.110:56532) 2024-09-15T03:58:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c659c640 1 -- 172.21.15.110:0/3489555560 learned_addr learned my addr 172.21.15.110:0/3489555560 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 10 monclient(hunting): picked mon.noname-c con 0x7fd1c81369e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f3051f54640 10 monclient: shutdown 2024-09-15T03:58:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f3051f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f3051f54640 1 -- 172.21.15.110:0/1106872379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c09ee80 msgr2=0x7f304c131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f3051f54640 1 --2- 172.21.15.110:0/1106872379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c09ee80 0x7f304c131b90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f304c0a2790 tx=0x7f30400090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 1 -- 172.21.15.110:0/3489555560 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c8099070 con 0x7fd1c809e770 2024-09-15T03:58:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f3051f54640 1 -- 172.21.15.110:0/1106872379 shutdown_connections 2024-09-15T03:58:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f3051f54640 1 --2- 172.21.15.110:0/1106872379 >> v2:172.21.15.110:3300/0 conn(0x7f304c1369e0 0x7f304c138dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f3051f54640 1 --2- 172.21.15.110:0/1106872379 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c09ee80 0x7f304c131b90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7f3051f54640 1 -- 172.21.15.110:0/1106872379 >> 172.21.15.110:0/1106872379 conn(0x7f304c06e860 msgr2=0x7f304c06ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 1 -- 172.21.15.110:0/3489555560 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c809b2e0 con 0x7fd1c80a1a10 2024-09-15T03:58:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 1 -- 172.21.15.110:0/3489555560 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c80983e0 con 0x7fd1c81369e0 2024-09-15T03:58:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1cc846640 10 monclient(hunting): authenticate will time out at 749.073833s 2024-09-15T03:58:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c659c640 10 monclient(hunting): get_auth_request con 0x7fd1c809e770 auth_method 0 2024-09-15T03:58:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c6d9d640 1 --2- 172.21.15.110:0/3489555560 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1c81369e0 0x7fd1c80974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c6d9d640 10 monclient(hunting): get_auth_request con 0x7fd1c81369e0 auth_method 0 2024-09-15T03:58:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.429+0000 7fd1c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7fd1c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7fd1c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7fd1c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7f3051f54640 1 -- 172.21.15.110:0/1106872379 shutdown_connections 2024-09-15T03:58:45.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7fd1c659c640 10 monclient(hunting): handle_auth_done global_id 4248 payload 824 2024-09-15T03:58:45.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7f3051f54640 1 -- 172.21.15.110:0/1106872379 wait complete. 2024-09-15T03:58:45.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7fd1c659c640 1 -- 172.21.15.110:0/3489555560 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1c81369e0 msgr2=0x7fd1c80974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7fd1c659c640 1 --2- 172.21.15.110:0/3489555560 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1c81369e0 0x7fd1c80974b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7fd1c659c640 1 -- 172.21.15.110:0/3489555560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1c80a1a10 msgr2=0x7fd1c8096f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7fd1c659c640 1 --2- 172.21.15.110:0/3489555560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1c80a1a10 0x7fd1c8096f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7fd1c659c640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7fd1c659c640 1 monclient: found mon.noname-a 2024-09-15T03:58:45.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7fd1c659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7fd1c659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.483 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7fd1c659c640 1 -- 172.21.15.110:0/3489555560 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b8009450 con 0x7fd1c809e770 2024-09-15T03:58:45.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7f3051f54640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.484 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7f3051f54640 10 monclient: monmap: 2024-09-15T03:58:45.484 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.484 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:45.484 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:45.484 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:45.484 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.484 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.484 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:45.484 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:45.484 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:45.484 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7f3d89d9b640 1 -- 172.21.15.110:0/922765203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d8c09ee80 msgr2=0x7f3d8c131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.430+0000 7f3d89d9b640 1 --2- 172.21.15.110:0/922765203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d8c09ee80 0x7f3d8c131b90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3d80009910 tx=0x7f3d800091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223ef2cf1c72c343 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f5d0500a640 1 -- 172.21.15.110:0/2595153138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d0006e610 msgr2=0x7f5d0006e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.485 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f5d0500a640 1 --2- 172.21.15.110:0/2595153138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d0006e610 0x7f5d0006e9e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009910 tx=0x7f5cf00091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c4a73c6e6c4fd41 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3051f54640 1 Processor -- start 2024-09-15T03:58:45.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3051f54640 1 -- start start 2024-09-15T03:58:45.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d8959a640 1 -- 172.21.15.110:0/922765203 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80032440 con 0x7f3d8c09ee80 2024-09-15T03:58:45.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d8959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d8959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:45.486 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3051f54640 10 monclient: init 2024-09-15T03:58:45.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d8959a640 10 monclient: dump: 2024-09-15T03:58:45.487 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.487 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.487 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.487 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.487 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.487 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.487 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.487 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.487 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.487 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d8959a640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d8959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d8959a640 1 -- 172.21.15.110:0/922765203 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d800325a0 con 0x7f3d8c09ee80 2024-09-15T03:58:45.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d8959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d8959a640 1 -- 172.21.15.110:0/922765203 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80032940 con 0x7f3d8c09ee80 2024-09-15T03:58:45.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d90c3f640 5 monclient: authenticate success, global_id 4245 2024-09-15T03:58:45.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d90c3f640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:45.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d90c3f640 4 set_mon_vals no callback set 2024-09-15T03:58:45.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d8959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d8959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:45.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.431+0000 7f3d8959a640 10 monclient: dump: 2024-09-15T03:58:45.488 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.488 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.488 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.488 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.489 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.489 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.489 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.489 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.489 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.489 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:58:45.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f304c09ee80 0x7f304c135070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d04809640 1 -- 172.21.15.110:0/2595153138 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0032440 con 0x7f5d0006e610 2024-09-15T03:58:45.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d04809640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d04809640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:45.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d04809640 10 monclient: dump: 2024-09-15T03:58:45.490 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.490 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.490 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.490 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.490 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.490 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.490 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.490 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.490 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.490 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d04809640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d04809640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d04809640 1 -- 172.21.15.110:0/2595153138 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf00325a0 con 0x7f5d0006e610 2024-09-15T03:58:45.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d07a6f640 5 monclient: authenticate success, global_id 4249 2024-09-15T03:58:45.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d04809640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d07a6f640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:45.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d04809640 1 -- 172.21.15.110:0/2595153138 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0032940 con 0x7f5d0006e610 2024-09-15T03:58:45.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d07a6f640 4 set_mon_vals no callback set 2024-09-15T03:58:45.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d04809640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d04809640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:45.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 10 monclient(hunting): picked mon.noname-c con 0x7f304c09ee80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c0a1a10 0x7f304c12fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f5d04809640 10 monclient: dump: 2024-09-15T03:58:45.492 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.492 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.492 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.492 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.492 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.492 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.492 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.492 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.493 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.493 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3d90c3f640 10 monclient: shutdown 2024-09-15T03:58:45.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 10 monclient(hunting): picked mon.noname-b con 0x7f304c0a1a10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:45.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3d90c3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3d90c3f640 1 -- 172.21.15.110:0/922765203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d8c09ee80 msgr2=0x7f3d8c131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3d90c3f640 1 --2- 172.21.15.110:0/922765203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d8c09ee80 0x7f3d8c131b90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3d80009910 tx=0x7f3d800091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f304c1369e0 0x7f304c1302b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f304c09ee80 0x7f304c135070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f304c09ee80 0x7f304c135070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55042/0 (socket says 172.21.15.110:55042) 2024-09-15T03:58:45.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 10 monclient(hunting): picked mon.noname-a con 0x7f304c1369e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:45.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304c133450 con 0x7f304c1369e0 2024-09-15T03:58:45.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304c0a0350 con 0x7f304c0a1a10 2024-09-15T03:58:45.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 1 -- 172.21.15.110:0/1685884872 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304c1349a0 con 0x7f304c09ee80 2024-09-15T03:58:45.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304b7fe640 1 -- 172.21.15.110:0/1685884872 learned_addr learned my addr 172.21.15.110:0/1685884872 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3051f54640 10 monclient(hunting): authenticate will time out at 749.076478s 2024-09-15T03:58:45.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1c659c640 1 -- 172.21.15.110:0/3489555560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1c809e770 msgr2=0x7fd1c8131060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1c659c640 1 --2- 172.21.15.110:0/3489555560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1c809e770 0x7fd1c8131060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8035040 tx=0x7fd1b8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c0a1a10 0x7f304c12fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304b7fe640 10 monclient(hunting): get_auth_request con 0x7f304c09ee80 auth_method 0 2024-09-15T03:58:45.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 10 monclient(hunting): get_auth_request con 0x7f304c0a1a10 auth_method 0 2024-09-15T03:58:45.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304bfff640 1 --2- 172.21.15.110:0/1685884872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f304c1369e0 0x7f304c1302b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304bfff640 10 monclient(hunting): get_auth_request con 0x7f304c1369e0 auth_method 0 2024-09-15T03:58:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3d90c3f640 1 -- 172.21.15.110:0/922765203 shutdown_connections 2024-09-15T03:58:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3d90c3f640 1 --2- 172.21.15.110:0/922765203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d8c09ee80 0x7f3d8c131b90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3d90c3f640 1 -- 172.21.15.110:0/922765203 >> 172.21.15.110:0/922765203 conn(0x7f3d8c06e860 msgr2=0x7f3d8c06ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f3d90c3f640 1 -- 172.21.15.110:0/922765203 shutdown_connections 2024-09-15T03:58:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1be7fc640 1 -- 172.21.15.110:0/3489555560 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8031d40 con 0x7fd1c809e770 2024-09-15T03:58:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1be7fc640 10 monclient: dump: 2024-09-15T03:58:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.499 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.500 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1be7fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1be7fc640 1 -- 172.21.15.110:0/3489555560 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b803d440 con 0x7fd1c809e770 2024-09-15T03:58:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1be7fc640 1 -- 172.21.15.110:0/3489555560 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b803d800 con 0x7fd1c809e770 2024-09-15T03:58:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1be7fc640 10 monclient: dump: 2024-09-15T03:58:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.500 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.501 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1c559a640 4 set_mon_vals no callback set 2024-09-15T03:58:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1cc846640 5 monclient: authenticate success, global_id 4248 2024-09-15T03:58:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1cc846640 10 monclient: _renew_subs 2024-09-15T03:58:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1cc846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7fd1cc846640 1 -- 172.21.15.110:0/3489555560 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c80a11c0 con 0x7fd1c809e770 2024-09-15T03:58:45.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7f5d07a6f640 10 monclient: shutdown 2024-09-15T03:58:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7f5d07a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7f5d07a6f640 1 -- 172.21.15.110:0/2595153138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d0006e610 msgr2=0x7f5d0006e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7f5d07a6f640 1 --2- 172.21.15.110:0/2595153138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d0006e610 0x7f5d0006e9e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009910 tx=0x7f5cf00091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7f3d90c3f640 1 -- 172.21.15.110:0/922765203 wait complete. 2024-09-15T03:58:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 10 monclient(hunting): handle_auth_done global_id 4252 payload 824 2024-09-15T03:58:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 1 -- 172.21.15.110:0/1685884872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f304c09ee80 msgr2=0x7f304c135070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 1 --2- 172.21.15.110:0/1685884872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f304c09ee80 0x7f304c135070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 1 -- 172.21.15.110:0/1685884872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f304c1369e0 msgr2=0x7f304c1302b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 1 --2- 172.21.15.110:0/1685884872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f304c1369e0 0x7f304c1302b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 1 monclient: found mon.noname-b 2024-09-15T03:58:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-09-15T03:58:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.432+0000 7f304affd640 1 -- 172.21.15.110:0/1685884872 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3040009390 con 0x7f304c0a1a10 2024-09-15T03:58:45.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7f304b7fe640 1 --2- 172.21.15.110:0/1685884872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f304c09ee80 0x7f304c135070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7fd1cc846640 10 monclient: _renew_subs 2024-09-15T03:58:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7fd1cc846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7fd1cc846640 1 -- 172.21.15.110:0/3489555560 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c809a620 con 0x7fd1c809e770 2024-09-15T03:58:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7f3d90c3f640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7f3d90c3f640 10 monclient: monmap: 2024-09-15T03:58:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:45.504 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.505 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.505 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:45.505 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:45.505 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:45.505 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7f5d07a6f640 1 -- 172.21.15.110:0/2595153138 shutdown_connections 2024-09-15T03:58:45.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7f5d07a6f640 1 --2- 172.21.15.110:0/2595153138 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d0006e610 0x7f5d0006e9e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7f5d07a6f640 1 --2- 172.21.15.110:0/2595153138 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d001368d0 0x7f5d00138cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7f5d07a6f640 1 -- 172.21.15.110:0/2595153138 >> 172.21.15.110:0/2595153138 conn(0x7f5d000a36b0 msgr2=0x7f5d0009f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7f5d07a6f640 1 -- 172.21.15.110:0/2595153138 shutdown_connections 2024-09-15T03:58:45.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.433+0000 7f5d07a6f640 1 -- 172.21.15.110:0/2595153138 wait complete. 2024-09-15T03:58:45.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7fd1be7fc640 1 -- 172.21.15.110:0/3489555560 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fd1b8039040 con 0x7fd1c809e770 2024-09-15T03:58:45.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7f3d90c3f640 1 Processor -- start 2024-09-15T03:58:45.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7fd1be7fc640 1 --2- 172.21.15.110:0/3489555560 >> v2:172.21.15.177:6812/28201 conn(0x7fd1a8038700 0x7fd1a803abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7f3d90c3f640 1 -- start start 2024-09-15T03:58:45.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7fd1be7fc640 1 -- 172.21.15.110:0/3489555560 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7fd1b806c430 con 0x7fd1c809e770 2024-09-15T03:58:45.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7f3d90c3f640 10 monclient: init 2024-09-15T03:58:45.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7f5d07a6f640 10 monclient: build_initial_monmap 2024-09-15T03:58:45.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7f5d07a6f640 10 monclient: monmap: 2024-09-15T03:58:45.507 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:45.507 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:45.508 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:45.508 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7fd1bffff640 1 --2- 172.21.15.110:0/3489555560 >> v2:172.21.15.177:6812/28201 conn(0x7fd1a8038700 0x7fd1a803abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7fd1bffff640 10 monclient: get_auth_request con 0x7fd1a8038700 auth_method 0 2024-09-15T03:58:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7fd1cc846640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7fd1cc846640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7f5d07a6f640 1 Processor -- start 2024-09-15T03:58:45.509 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7fd1cc846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7fd1cc846640 1 -- 172.21.15.110:0/3489555560 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c809a990 con 0x7fd1c809e770 2024-09-15T03:58:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7f5d07a6f640 1 -- start start 2024-09-15T03:58:45.510 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.434+0000 7f5d07a6f640 10 monclient: init 2024-09-15T03:58:45.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.511 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d8c09ee80 0x7f3d8c128ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 10 monclient(hunting): picked mon.noname-c con 0x7f3d8c09ee80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d8c0a1a10 0x7f3d8c130c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 10 monclient(hunting): picked mon.noname-a con 0x7f3d8c0a1a10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d8c09ee80 0x7f3d8c128ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d8c09ee80 0x7f3d8c128ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55058/0 (socket says 172.21.15.110:55058) 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d8c1369e0 0x7f3d8c133a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 1 -- 172.21.15.110:0/3699539543 learned_addr learned my addr 172.21.15.110:0/3699539543 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 10 monclient(hunting): picked mon.noname-b con 0x7f3d8c1369e0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 1 -- 172.21.15.110:0/3699539543 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c0a1230 con 0x7f3d8c0a1a10 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 1 -- 172.21.15.110:0/3699539543 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c06ae00 con 0x7f3d8c1369e0 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.512 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 1 -- 172.21.15.110:0/3699539543 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c06b7e0 con 0x7f3d8c09ee80 2024-09-15T03:58:45.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d90c3f640 10 monclient(hunting): authenticate will time out at 749.079288s 2024-09-15T03:58:45.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 10 monclient(hunting): get_auth_request con 0x7f3d8c09ee80 auth_method 0 2024-09-15T03:58:45.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d89d9b640 1 --2- 172.21.15.110:0/3699539543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d8c0a1a10 0x7f3d8c130c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d89d9b640 10 monclient(hunting): get_auth_request con 0x7f3d8c0a1a10 auth_method 0 2024-09-15T03:58:45.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d89d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.513 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f304affd640 1 -- 172.21.15.110:0/1685884872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c0a1a10 msgr2=0x7f304c12fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f304affd640 1 --2- 172.21.15.110:0/1685884872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c0a1a10 0x7f304c12fd70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f304c09cae0 tx=0x7f3040032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:45.514 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:58:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d89d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d89d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d89d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 10 monclient(hunting): handle_auth_done global_id 4271 payload 824 2024-09-15T03:58:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d0006e610 0x7f5d00133dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.515 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 1 -- 172.21.15.110:0/3699539543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d8c1369e0 msgr2=0x7f3d8c133a40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T03:58:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 1 --2- 172.21.15.110:0/3699539543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d8c1369e0 0x7f3d8c133a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 1 -- 172.21.15.110:0/3699539543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d8c0a1a10 msgr2=0x7f3d8c130c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 1 --2- 172.21.15.110:0/3699539543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d8c0a1a10 0x7f3d8c130c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.516 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 10 monclient(hunting): picked mon.noname-c con 0x7f5d0006e610 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d001315a0 0x7f5d00134310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 10 monclient(hunting): picked mon.noname-b con 0x7f5d001315a0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d001368d0 0x7f5d00134850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f30337fe640 1 -- 172.21.15.110:0/1685884872 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304003c900 con 0x7f304c0a1a10 2024-09-15T03:58:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f30337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.517 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f30337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 10 monclient(hunting): picked mon.noname-a con 0x7f5d001368d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f30337fe640 10 monclient: dump: 2024-09-15T03:58:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.518 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.519 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f30337fe640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f30337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f30337fe640 1 -- 172.21.15.110:0/1685884872 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304003ca80 con 0x7f304c0a1a10 2024-09-15T03:58:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f30337fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f30337fe640 1 -- 172.21.15.110:0/1685884872 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304003ce40 con 0x7f304c0a1a10 2024-09-15T03:58:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f30337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f30337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:45.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f30337fe640 10 monclient: dump: 2024-09-15T03:58:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.520 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3051f54640 5 monclient: authenticate success, global_id 4252 2024-09-15T03:58:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3051f54640 10 monclient: _renew_subs 2024-09-15T03:58:45.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3051f54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:45.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3051f54640 1 -- 172.21.15.110:0/1685884872 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f304c133640 con 0x7f304c0a1a10 2024-09-15T03:58:45.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 1 monclient: found mon.noname-c 2024-09-15T03:58:45.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3d8a59c640 1 -- 172.21.15.110:0/3699539543 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d80009450 con 0x7f3d8c09ee80 2024-09-15T03:58:45.521 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0011a420 con 0x7f5d001368d0 2024-09-15T03:58:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0006f100 con 0x7f5d001315a0 2024-09-15T03:58:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0006d710 con 0x7f5d0006e610 2024-09-15T03:58:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f5d07a6f640 10 monclient(hunting): authenticate will time out at 749.079922s 2024-09-15T03:58:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0500a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d001315a0 0x7f5d00134310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d001368d0 0x7f5d00134850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.522 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.435+0000 7f3049ffb640 4 set_mon_vals no callback set 2024-09-15T03:58:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0500a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d001315a0 0x7f5d00134310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.110:59858/0 (socket says 172.21.15.110:59858) 2024-09-15T03:58:45.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d001368d0 0x7f5d00134850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56562/0 (socket says 172.21.15.110:56562) 2024-09-15T03:58:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 1 -- 172.21.15.110:0/1395567377 learned_addr learned my addr 172.21.15.110:0/1395567377 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 10 monclient(hunting): get_auth_request con 0x7f5d001368d0 auth_method 0 2024-09-15T03:58:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0500a640 10 monclient(hunting): get_auth_request con 0x7f5d001315a0 auth_method 0 2024-09-15T03:58:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0500a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0500a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0580b640 1 --2- 172.21.15.110:0/1395567377 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d0006e610 0x7f5d00133dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0500a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f3051f54640 10 monclient: _renew_subs 2024-09-15T03:58:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f3051f54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f3051f54640 1 -- 172.21.15.110:0/1685884872 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f304c1355b0 con 0x7f304c0a1a10 2024-09-15T03:58:45.525 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0580b640 10 monclient(hunting): get_auth_request con 0x7f5d0006e610 auth_method 0 2024-09-15T03:58:45.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0580b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:45.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.437+0000 7f30337fe640 1 -- 172.21.15.110:0/1685884872 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f3040039050 con 0x7f304c0a1a10 2024-09-15T03:58:45.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0580b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:45.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0580b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:45.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0580b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:45.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0580b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:45.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0580b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:45.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.437+0000 7fd1bffff640 1 --2- 172.21.15.110:0/3489555560 >> v2:172.21.15.177:6812/28201 conn(0x7fd1a8038700 0x7fd1a803abb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0007fd0 tx=0x7fd1b0007a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.437+0000 7f30337fe640 1 --2- 172.21.15.110:0/1685884872 >> v2:172.21.15.177:6812/28201 conn(0x7f3038038750 0x7f303803ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 10 monclient(hunting): handle_auth_done global_id 4257 payload 824 2024-09-15T03:58:45.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 1 -- 172.21.15.110:0/1395567377 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d0006e610 msgr2=0x7f5d00133dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.437+0000 7f30337fe640 1 -- 172.21.15.110:0/1685884872 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f304006bc90 con 0x7f304c0a1a10 2024-09-15T03:58:45.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 1 --2- 172.21.15.110:0/1395567377 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d0006e610 0x7f5d00133dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 1 -- 172.21.15.110:0/1395567377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d001315a0 msgr2=0x7f5d00134310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.437+0000 7f3051f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 1 --2- 172.21.15.110:0/1395567377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d001315a0 0x7f5d00134310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 10 monclient: _finish_hunting 0 2024-09-15T03:58:45.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 1 monclient: found mon.noname-a 2024-09-15T03:58:45.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:45.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.436+0000 7f5d0600c640 1 -- 172.21.15.110:0/1395567377 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf0009450 con 0x7f5d001368d0 2024-09-15T03:58:45.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.437+0000 7f3051f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.437+0000 7f3051f54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:45.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.437+0000 7f304b7fe640 1 --2- 172.21.15.110:0/1685884872 >> v2:172.21.15.177:6812/28201 conn(0x7f3038038750 0x7f303803ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.437+0000 7f3051f54640 1 -- 172.21.15.110:0/1685884872 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f304c1347d0 con 0x7f304c0a1a10 2024-09-15T03:58:45.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.437+0000 7f304b7fe640 10 monclient: get_auth_request con 0x7f3038038750 auth_method 0 2024-09-15T03:58:45.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d8a59c640 1 -- 172.21.15.110:0/3699539543 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d8c09ee80 msgr2=0x7f3d8c128ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d8a59c640 1 --2- 172.21.15.110:0/3699539543 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d8c09ee80 0x7f3d8c128ee0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c00b520 tx=0x7f3d7c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d7a7fc640 1 -- 172.21.15.110:0/3699539543 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c00f720 con 0x7f3d8c09ee80 2024-09-15T03:58:45.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:45.532 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d7a7fc640 10 monclient: dump: 2024-09-15T03:58:45.532 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.532 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.532 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.532 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.532 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.532 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.532 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.533 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.533 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.533 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d7a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d7a7fc640 1 -- 172.21.15.110:0/3699539543 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7c00f8a0 con 0x7f3d8c09ee80 2024-09-15T03:58:45.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d7a7fc640 1 -- 172.21.15.110:0/3699539543 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c00fc60 con 0x7f3d8c09ee80 2024-09-15T03:58:45.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d90c3f640 5 monclient: authenticate success, global_id 4271 2024-09-15T03:58:45.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d90c3f640 10 monclient: _renew_subs 2024-09-15T03:58:45.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d90c3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d90c3f640 1 -- 172.21.15.110:0/3699539543 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8c0a2db0 con 0x7f3d8c09ee80 2024-09-15T03:58:45.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d7a7fc640 10 monclient: dump: 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d88d99640 4 set_mon_vals no callback set 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d90c3f640 10 monclient: _renew_subs 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d90c3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f3d90c3f640 1 -- 172.21.15.110:0/3699539543 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8c135e80 con 0x7f3d8c09ee80 2024-09-15T03:58:45.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f5d0600c640 1 -- 172.21.15.110:0/1395567377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d001368d0 msgr2=0x7f5d00134850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:45.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.438+0000 7f5d0600c640 1 --2- 172.21.15.110:0/1395567377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d001368d0 0x7f5d00134850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8009df0 tx=0x7f5cf800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.439+0000 7f3d90c3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.439+0000 7f3d90c3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.439+0000 7f3d90c3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.439+0000 7f3d90c3f640 1 -- 172.21.15.110:0/3699539543 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8c06b320 con 0x7f3d8c09ee80 2024-09-15T03:58:45.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f3d7a7fc640 1 -- 172.21.15.110:0/3699539543 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f3d7c011610 con 0x7f3d8c09ee80 2024-09-15T03:58:45.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f304b7fe640 1 --2- 172.21.15.110:0/1685884872 >> v2:172.21.15.177:6812/28201 conn(0x7f3038038750 0x7f303803ac00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f303c006fd0 tx=0x7f303c004960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5cf5ffb640 1 -- 172.21.15.110:0/1395567377 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf80179c0 con 0x7f5d001368d0 2024-09-15T03:58:45.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f3d7a7fc640 1 --2- 172.21.15.110:0/3699539543 >> v2:172.21.15.177:6812/28201 conn(0x7f3d6c038750 0x7f3d6c03ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5cf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7fd1be7fc640 1 -- 172.21.15.110:0/3489555560 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd1b8042020 con 0x7fd1c809e770 2024-09-15T03:58:45.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7fd1be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7fd1be7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:45.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f3d7a7fc640 1 -- 172.21.15.110:0/3699539543 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f3d7c053350 con 0x7f3d8c09ee80 2024-09-15T03:58:45.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f3d89d9b640 1 --2- 172.21.15.110:0/3699539543 >> v2:172.21.15.177:6812/28201 conn(0x7f3d6c038750 0x7f3d6c03ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f3d89d9b640 10 monclient: get_auth_request con 0x7f3d6c038750 auth_method 0 2024-09-15T03:58:45.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5cf5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:45.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5cf5ffb640 10 monclient: dump: 2024-09-15T03:58:45.539 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.540 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5cf5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5cf5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5cf5ffb640 1 -- 172.21.15.110:0/1395567377 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf800f040 con 0x7f5d001368d0 2024-09-15T03:58:45.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5d07a6f640 5 monclient: authenticate success, global_id 4257 2024-09-15T03:58:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5d07a6f640 10 monclient: _renew_subs 2024-09-15T03:58:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5d07a6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5d07a6f640 1 -- 172.21.15.110:0/1395567377 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d00119350 con 0x7f5d001368d0 2024-09-15T03:58:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5d07a6f640 10 monclient: _renew_subs 2024-09-15T03:58:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5d07a6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5d07a6f640 1 -- 172.21.15.110:0/1395567377 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d001311a0 con 0x7f5d001368d0 2024-09-15T03:58:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5cf5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5cf5ffb640 1 -- 172.21.15.110:0/1395567377 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8017d70 con 0x7f5d001368d0 2024-09-15T03:58:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5cf5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5cf5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5cf5ffb640 10 monclient: dump: 2024-09-15T03:58:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:45.541 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:45.542 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.440+0000 7f5cf77fe640 4 set_mon_vals no callback set 2024-09-15T03:58:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.441+0000 7f5cf5ffb640 1 -- 172.21.15.110:0/1395567377 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f5cf80283b0 con 0x7f5d001368d0 2024-09-15T03:58:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.441+0000 7f5cf5ffb640 1 --2- 172.21.15.110:0/1395567377 >> v2:172.21.15.177:6812/28201 conn(0x7f5ce4038750 0x7f5ce403ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.441+0000 7f5cf5ffb640 1 -- 172.21.15.110:0/1395567377 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f5cf8054140 con 0x7f5d001368d0 2024-09-15T03:58:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.442+0000 7f5d07a6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.442+0000 7f5d07a6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.442+0000 7f5d07a6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.442+0000 7f5d07a6f640 1 -- 172.21.15.110:0/1395567377 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce0005010 con 0x7f5d001368d0 2024-09-15T03:58:45.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.442+0000 7f30337fe640 1 -- 172.21.15.110:0/1685884872 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f304c1347d0 con 0x7f304c0a1a10 2024-09-15T03:58:45.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.442+0000 7f30337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.442+0000 7f30337fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:45.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.443+0000 7f3d89d9b640 1 --2- 172.21.15.110:0/3699539543 >> v2:172.21.15.177:6812/28201 conn(0x7f3d6c038750 0x7f3d6c03ac00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3d80031000 tx=0x7f3d80032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.443+0000 7f5d0580b640 1 --2- 172.21.15.110:0/1395567377 >> v2:172.21.15.177:6812/28201 conn(0x7f5ce4038750 0x7f5ce403ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:45.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.443+0000 7f5d0580b640 10 monclient: get_auth_request con 0x7f5ce4038750 auth_method 0 2024-09-15T03:58:45.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.445+0000 7f5d0580b640 1 --2- 172.21.15.110:0/1395567377 >> v2:172.21.15.177:6812/28201 conn(0x7f5ce4038750 0x7f5ce403ac00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc009990 tx=0x7f5cfc006f30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:45.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.447+0000 7f3d7a7fc640 1 -- 172.21.15.110:0/3699539543 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3d8c06b320 con 0x7f3d8c09ee80 2024-09-15T03:58:45.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.447+0000 7f3d7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.447+0000 7f3d7a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:45.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.451+0000 7f5cf5ffb640 1 -- 172.21.15.110:0/1395567377 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5cf8024080 con 0x7f5d001368d0 2024-09-15T03:58:45.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.451+0000 7f5cf5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:45.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.451+0000 7f5cf5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:45.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.606+0000 7f3051f54640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:58:45.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.606+0000 7f3051f54640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:58:45.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.606+0000 7f3051f54640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:45.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.606+0000 7f3051f54640 1 -- 172.21.15.110:0/1685884872 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f304c1347d0 con 0x7f304c0a1a10 2024-09-15T03:58:45.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.606+0000 7f30337fe640 1 -- 172.21.15.110:0/1685884872 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f304c1347d0 con 0x7f304c0a1a10 2024-09-15T03:58:45.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.606+0000 7f30337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:58:45.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.606+0000 7f30337fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:45.608 INFO:teuthology.orchestra.run.smithi110.stdout:47244640258 2024-09-15T03:58:45.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3051f54640 1 -- 172.21.15.110:0/1685884872 >> v2:172.21.15.177:6812/28201 conn(0x7f3038038750 msgr2=0x7f303803ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3051f54640 1 --2- 172.21.15.110:0/1685884872 >> v2:172.21.15.177:6812/28201 conn(0x7f3038038750 0x7f303803ac00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f303c006fd0 tx=0x7f303c004960 comp rx=0 tx=0).stop 2024-09-15T03:58:45.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3051f54640 10 monclient: shutdown 2024-09-15T03:58:45.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3d90c3f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:58:45.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3d90c3f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:58:45.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3d90c3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3051f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3051f54640 1 -- 172.21.15.110:0/1685884872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c0a1a10 msgr2=0x7f304c12fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.610 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3051f54640 1 --2- 172.21.15.110:0/1685884872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c0a1a10 0x7f304c12fd70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f304c09cae0 tx=0x7f3040032000 comp rx=0 tx=0).stop 2024-09-15T03:58:45.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3d90c3f640 1 -- 172.21.15.110:0/3699539543 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3d8c06b320 con 0x7f3d8c09ee80 2024-09-15T03:58:45.611 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3051f54640 1 -- 172.21.15.110:0/1685884872 shutdown_connections 2024-09-15T03:58:45.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3051f54640 1 --2- 172.21.15.110:0/1685884872 >> v2:172.21.15.177:6812/28201 conn(0x7f3038038750 0x7f303803ac00 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f303c006fd0 tx=0x7f303c004960 comp rx=0 tx=0).stop 2024-09-15T03:58:45.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3051f54640 1 --2- 172.21.15.110:0/1685884872 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f304c1369e0 0x7f304c1302b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3051f54640 1 --2- 172.21.15.110:0/1685884872 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f304c0a1a10 0x7f304c12fd70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3051f54640 1 --2- 172.21.15.110:0/1685884872 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f304c09ee80 0x7f304c135070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3051f54640 1 -- 172.21.15.110:0/1685884872 >> 172.21.15.110:0/1685884872 conn(0x7f304c06e860 msgr2=0x7f304c09d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3051f54640 1 -- 172.21.15.110:0/1685884872 shutdown_connections 2024-09-15T03:58:45.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3051f54640 1 -- 172.21.15.110:0/1685884872 wait complete. 2024-09-15T03:58:45.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3d7a7fc640 1 -- 172.21.15.110:0/3699539543 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3d8c06b320 con 0x7f3d8c09ee80 2024-09-15T03:58:45.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3d7a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:58:45.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.609+0000 7f3d7a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:45.613 INFO:teuthology.orchestra.run.smithi110.stdout:47244640258 2024-09-15T03:58:45.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.611+0000 7f3d90c3f640 1 -- 172.21.15.110:0/3699539543 >> v2:172.21.15.177:6812/28201 conn(0x7f3d6c038750 msgr2=0x7f3d6c03ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.611+0000 7f3d90c3f640 1 --2- 172.21.15.110:0/3699539543 >> v2:172.21.15.177:6812/28201 conn(0x7f3d6c038750 0x7f3d6c03ac00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3d80031000 tx=0x7f3d80032000 comp rx=0 tx=0).stop 2024-09-15T03:58:45.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.611+0000 7f3d90c3f640 10 monclient: shutdown 2024-09-15T03:58:45.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.611+0000 7f3d90c3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.611+0000 7f3d90c3f640 1 -- 172.21.15.110:0/3699539543 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d8c09ee80 msgr2=0x7f3d8c128ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.611+0000 7f3d90c3f640 1 --2- 172.21.15.110:0/3699539543 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d8c09ee80 0x7f3d8c128ee0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c00b520 tx=0x7f3d7c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.611+0000 7f3d90c3f640 1 -- 172.21.15.110:0/3699539543 shutdown_connections 2024-09-15T03:58:45.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.611+0000 7f3d90c3f640 1 --2- 172.21.15.110:0/3699539543 >> v2:172.21.15.177:6812/28201 conn(0x7f3d6c038750 0x7f3d6c03ac00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.611+0000 7f3d90c3f640 1 --2- 172.21.15.110:0/3699539543 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3d8c1369e0 0x7f3d8c133a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.611+0000 7f3d90c3f640 1 --2- 172.21.15.110:0/3699539543 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3d8c0a1a10 0x7f3d8c130c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.611+0000 7f3d90c3f640 1 --2- 172.21.15.110:0/3699539543 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3d8c09ee80 0x7f3d8c128ee0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.612+0000 7f3d90c3f640 1 -- 172.21.15.110:0/3699539543 >> 172.21.15.110:0/3699539543 conn(0x7f3d8c06e860 msgr2=0x7f3d8c12b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.612+0000 7f3d90c3f640 1 -- 172.21.15.110:0/3699539543 shutdown_connections 2024-09-15T03:58:45.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.613+0000 7f3d90c3f640 1 -- 172.21.15.110:0/3699539543 wait complete. 2024-09-15T03:58:45.624 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2024-09-15T03:58:45.626 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.5 2024-09-15T03:58:45.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.737+0000 7fd1cc846640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:58:45.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.737+0000 7fd1cc846640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:58:45.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.737+0000 7fd1cc846640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.737+0000 7fd1cc846640 1 -- 172.21.15.110:0/3489555560 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd1c80647e0 con 0x7fd1c809e770 2024-09-15T03:58:45.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.737+0000 7fd1be7fc640 1 -- 172.21.15.110:0/3489555560 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd1b8037070 con 0x7fd1c809e770 2024-09-15T03:58:45.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.737+0000 7fd1be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:58:45.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.737+0000 7fd1be7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:45.739 INFO:teuthology.orchestra.run.smithi110.stdout:47244640258 2024-09-15T03:58:45.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.739+0000 7fd1cc846640 1 -- 172.21.15.110:0/3489555560 >> v2:172.21.15.177:6812/28201 conn(0x7fd1a8038700 msgr2=0x7fd1a803abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.739+0000 7fd1cc846640 1 --2- 172.21.15.110:0/3489555560 >> v2:172.21.15.177:6812/28201 conn(0x7fd1a8038700 0x7fd1a803abb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0007fd0 tx=0x7fd1b0007a90 comp rx=0 tx=0).stop 2024-09-15T03:58:45.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.739+0000 7fd1cc846640 10 monclient: shutdown 2024-09-15T03:58:45.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.739+0000 7fd1cc846640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.739+0000 7fd1cc846640 1 -- 172.21.15.110:0/3489555560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1c809e770 msgr2=0x7fd1c8131060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.739+0000 7fd1cc846640 1 --2- 172.21.15.110:0/3489555560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1c809e770 0x7fd1c8131060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8035040 tx=0x7fd1b8032000 comp rx=0 tx=0).stop 2024-09-15T03:58:45.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.739+0000 7fd1cc846640 1 -- 172.21.15.110:0/3489555560 shutdown_connections 2024-09-15T03:58:45.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.739+0000 7fd1cc846640 1 --2- 172.21.15.110:0/3489555560 >> v2:172.21.15.177:6812/28201 conn(0x7fd1a8038700 0x7fd1a803abb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.739+0000 7fd1cc846640 1 --2- 172.21.15.110:0/3489555560 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd1c81369e0 0x7fd1c80974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.739+0000 7fd1cc846640 1 --2- 172.21.15.110:0/3489555560 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd1c80a1a10 0x7fd1c8096f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.739+0000 7fd1cc846640 1 --2- 172.21.15.110:0/3489555560 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd1c809e770 0x7fd1c8131060 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.739+0000 7fd1cc846640 1 -- 172.21.15.110:0/3489555560 >> 172.21.15.110:0/3489555560 conn(0x7fd1c806e860 msgr2=0x7fd1c80a4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.740+0000 7fd1cc846640 1 -- 172.21.15.110:0/3489555560 shutdown_connections 2024-09-15T03:58:45.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.740+0000 7fd1cc846640 1 -- 172.21.15.110:0/3489555560 wait complete. 2024-09-15T03:58:45.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.743+0000 7f1120c4f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:58:45.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.743+0000 7f1120c4f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:58:45.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.743+0000 7f5d07a6f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:58:45.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.743+0000 7f5d07a6f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:58:45.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.743+0000 7f5d07a6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:45.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.743+0000 7f1120c4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.743+0000 7f1120c4f640 1 -- 172.21.15.110:0/3512942837 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f111c135e00 con 0x7f111c1369e0 2024-09-15T03:58:45.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.743+0000 7f5d07a6f640 1 -- 172.21.15.110:0/1395567377 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f5ce00052c0 con 0x7f5d001368d0 2024-09-15T03:58:45.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.743+0000 7f5cf5ffb640 1 -- 172.21.15.110:0/1395567377 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5cf801ca80 con 0x7f5d001368d0 2024-09-15T03:58:45.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.743+0000 7f110a7fc640 1 -- 172.21.15.110:0/3512942837 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1114010af0 con 0x7f111c1369e0 2024-09-15T03:58:45.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.743+0000 7f110a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:58:45.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.743+0000 7f110a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:45.746 INFO:teuthology.orchestra.run.smithi110.stdout:47244640258 2024-09-15T03:58:45.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.743+0000 7f5cf5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:58:45.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.743+0000 7f5cf5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:45.746 INFO:teuthology.orchestra.run.smithi110.stdout:47244640258 2024-09-15T03:58:45.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.745+0000 7f1120c4f640 1 -- 172.21.15.110:0/3512942837 >> v2:172.21.15.177:6812/28201 conn(0x7f10fc038750 msgr2=0x7f10fc03ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.745+0000 7f1120c4f640 1 --2- 172.21.15.110:0/3512942837 >> v2:172.21.15.177:6812/28201 conn(0x7f10fc038750 0x7f10fc03ac00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f111c0a0c30 tx=0x7f110c032000 comp rx=0 tx=0).stop 2024-09-15T03:58:45.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.745+0000 7f1120c4f640 10 monclient: shutdown 2024-09-15T03:58:45.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.745+0000 7f1120c4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.745+0000 7f1120c4f640 1 -- 172.21.15.110:0/3512942837 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c1369e0 msgr2=0x7f111c1344e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.745+0000 7f1120c4f640 1 --2- 172.21.15.110:0/3512942837 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c1369e0 0x7f111c1344e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f111400ccc0 tx=0x7f1114005370 comp rx=0 tx=0).stop 2024-09-15T03:58:45.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.745+0000 7f1120c4f640 1 -- 172.21.15.110:0/3512942837 shutdown_connections 2024-09-15T03:58:45.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.745+0000 7f1120c4f640 1 --2- 172.21.15.110:0/3512942837 >> v2:172.21.15.177:6812/28201 conn(0x7f10fc038750 0x7f10fc03ac00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.745+0000 7f1120c4f640 1 --2- 172.21.15.110:0/3512942837 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f111c1369e0 0x7f111c1344e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.745+0000 7f1120c4f640 1 --2- 172.21.15.110:0/3512942837 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f111c0a1a10 0x7f111c133fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.745+0000 7f1120c4f640 1 --2- 172.21.15.110:0/3512942837 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f111c09e770 0x7f111c133a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.745+0000 7f1120c4f640 1 -- 172.21.15.110:0/3512942837 >> 172.21.15.110:0/3512942837 conn(0x7f111c06e860 msgr2=0x7f111c0a2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.745+0000 7f5d07a6f640 1 -- 172.21.15.110:0/1395567377 >> v2:172.21.15.177:6812/28201 conn(0x7f5ce4038750 msgr2=0x7f5ce403ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.745+0000 7f5d07a6f640 1 --2- 172.21.15.110:0/1395567377 >> v2:172.21.15.177:6812/28201 conn(0x7f5ce4038750 0x7f5ce403ac00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc009990 tx=0x7f5cfc006f30 comp rx=0 tx=0).stop 2024-09-15T03:58:45.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.746+0000 7f5d07a6f640 10 monclient: shutdown 2024-09-15T03:58:45.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.746+0000 7f5d07a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.746+0000 7f5d07a6f640 1 -- 172.21.15.110:0/1395567377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d001368d0 msgr2=0x7f5d00134850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.746+0000 7f5d07a6f640 1 --2- 172.21.15.110:0/1395567377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d001368d0 0x7f5d00134850 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5cf8009df0 tx=0x7f5cf800b040 comp rx=0 tx=0).stop 2024-09-15T03:58:45.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.746+0000 7f1120c4f640 1 -- 172.21.15.110:0/3512942837 shutdown_connections 2024-09-15T03:58:45.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.746+0000 7f1120c4f640 1 -- 172.21.15.110:0/3512942837 wait complete. 2024-09-15T03:58:45.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.746+0000 7f5d07a6f640 1 -- 172.21.15.110:0/1395567377 shutdown_connections 2024-09-15T03:58:45.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.746+0000 7f5d07a6f640 1 --2- 172.21.15.110:0/1395567377 >> v2:172.21.15.177:6812/28201 conn(0x7f5ce4038750 0x7f5ce403ac00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.746+0000 7f5d07a6f640 1 --2- 172.21.15.110:0/1395567377 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d001368d0 0x7f5d00134850 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.746+0000 7f5d07a6f640 1 --2- 172.21.15.110:0/1395567377 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d001315a0 0x7f5d00134310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.746+0000 7f5d07a6f640 1 --2- 172.21.15.110:0/1395567377 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d0006e610 0x7f5d00133dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.746+0000 7f5d07a6f640 1 -- 172.21.15.110:0/1395567377 >> 172.21.15.110:0/1395567377 conn(0x7f5d000a36b0 msgr2=0x7f5d000a1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.746+0000 7f5d07a6f640 1 -- 172.21.15.110:0/1395567377 shutdown_connections 2024-09-15T03:58:45.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.746+0000 7f5d07a6f640 1 -- 172.21.15.110:0/1395567377 wait complete. 2024-09-15T03:58:45.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.748+0000 7f03237fe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:58:45.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.748+0000 7f03237fe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:58:45.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.748+0000 7f03237fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:45.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.748+0000 7f03237fe640 1 -- 172.21.15.110:0/444383082 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f03380a1250 con 0x7f0338131a80 2024-09-15T03:58:45.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.748+0000 7f032dffb640 1 -- 172.21.15.110:0/444383082 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f03380a1250 con 0x7f0338131a80 2024-09-15T03:58:45.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.748+0000 7f032dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:58:45.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.748+0000 7f032dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:45.752 INFO:teuthology.orchestra.run.smithi110.stdout:47244640258 2024-09-15T03:58:45.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.751+0000 7f03237fe640 1 -- 172.21.15.110:0/444383082 >> v2:172.21.15.177:6812/28201 conn(0x7f031c038750 msgr2=0x7f031c03ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.751+0000 7f03237fe640 1 --2- 172.21.15.110:0/444383082 >> v2:172.21.15.177:6812/28201 conn(0x7f031c038750 0x7f031c03ac00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0334007fd0 tx=0x7f0334009040 comp rx=0 tx=0).stop 2024-09-15T03:58:45.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.751+0000 7f03237fe640 10 monclient: shutdown 2024-09-15T03:58:45.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.751+0000 7f03237fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:45.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.751+0000 7f03237fe640 1 -- 172.21.15.110:0/444383082 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0338131a80 msgr2=0x7f033806a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:45.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.751+0000 7f03237fe640 1 --2- 172.21.15.110:0/444383082 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0338131a80 0x7f033806a990 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0330034000 tx=0x7f0330037000 comp rx=0 tx=0).stop 2024-09-15T03:58:45.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.751+0000 7f03237fe640 1 -- 172.21.15.110:0/444383082 shutdown_connections 2024-09-15T03:58:45.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.751+0000 7f03237fe640 1 --2- 172.21.15.110:0/444383082 >> v2:172.21.15.177:6812/28201 conn(0x7f031c038750 0x7f031c03ac00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.751+0000 7f03237fe640 1 --2- 172.21.15.110:0/444383082 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0338131a80 0x7f033806a990 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.751+0000 7f03237fe640 1 --2- 172.21.15.110:0/444383082 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03380a1cf0 0x7f033806a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.751+0000 7f03237fe640 1 --2- 172.21.15.110:0/444383082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f033809ed70 0x7f033806db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:45.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.751+0000 7f03237fe640 1 -- 172.21.15.110:0/444383082 >> 172.21.15.110:0/444383082 conn(0x7f0338062f70 msgr2=0x7f0338134620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:45.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.751+0000 7f03237fe640 1 -- 172.21.15.110:0/444383082 shutdown_connections 2024-09-15T03:58:45.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:45.751+0000 7f03237fe640 1 -- 172.21.15.110:0/444383082 wait complete. 2024-09-15T03:58:45.754 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.0 2024-09-15T03:58:45.760 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.4 2024-09-15T03:58:45.761 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.2 2024-09-15T03:58:45.765 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.1 2024-09-15T03:58:46.625 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:58:46.627 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T03:58:46.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.725+0000 7f59d4c89640 10 monclient: get_monmap_and_config 2024-09-15T03:58:46.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.725+0000 7f59d4c89640 10 monclient: build_initial_monmap 2024-09-15T03:58:46.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.725+0000 7f59d4c89640 10 monclient: monmap: 2024-09-15T03:58:46.727 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:46.727 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.728 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:46.727579+0000 2024-09-15T03:58:46.728 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:46.727579+0000 2024-09-15T03:58:46.728 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:46.728 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.728 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:46.728 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:46.728 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:46.728 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:46.728 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:46.728 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.725+0000 7f6a1fd1d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:46.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.725+0000 7f6a1fd1d640 10 monclient: build_initial_monmap 2024-09-15T03:58:46.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.725+0000 7f6a1fd1d640 10 monclient: monmap: 2024-09-15T03:58:46.729 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:46.729 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.729 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:46.727670+0000 2024-09-15T03:58:46.729 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:46.727670+0000 2024-09-15T03:58:46.729 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:46.729 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.729 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:46.730 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:46.730 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:46.730 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:46.730 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:46.730 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 1 Processor -- start 2024-09-15T03:58:46.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 1 -- start start 2024-09-15T03:58:46.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 10 monclient: init 2024-09-15T03:58:46.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f6a1fd1d640 1 Processor -- start 2024-09-15T03:58:46.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f6a1fd1d640 1 -- start start 2024-09-15T03:58:46.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f6a1fd1d640 10 monclient: init 2024-09-15T03:58:46.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:46.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-15T03:58:46.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59d00a1a10 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:46.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59d009ee80 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:46.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f59d0131b90 0x7f59d013a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 10 monclient(hunting): picked mon.noname-c con 0x7f59d0131b90 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:46.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d009f2c0 con 0x7f59d00a1a10 2024-09-15T03:58:46.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d00a0850 con 0x7f59d009ee80 2024-09-15T03:58:46.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d009c9a0 con 0x7f59d0131b90 2024-09-15T03:58:46.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:46.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59d4c89640 10 monclient(hunting): authenticate will time out at 750.371889s 2024-09-15T03:58:46.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59ce59c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f59d00a1a10 0x7f59d009e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33834/0 (socket says 172.21.15.110:33834) 2024-09-15T03:58:46.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.726+0000 7f59ce59c640 1 -- 172.21.15.110:0/2816115934 learned_addr learned my addr 172.21.15.110:0/2816115934 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:46.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1fd1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:46.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1fd1d640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T03:58:46.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1fd1d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6a180a24a0 0x7f6a1809ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1fd1d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a180a24a0 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:46.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1fd1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a18067130 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:46.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1fd1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a18067ca0 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:46.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1fd1d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a1809c800 con 0x7f6a18067130 2024-09-15T03:58:46.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1fd1d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a1809e2f0 con 0x7f6a18067ca0 2024-09-15T03:58:46.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1fd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1fd1d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1809d530 con 0x7f6a180a24a0 2024-09-15T03:58:46.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1fd1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:46.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1fd1d640 10 monclient(hunting): authenticate will time out at 750.372112s 2024-09-15T03:58:46.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f59ced9d640 1 --2- 172.21.15.110:0/2816115934 >> v2:172.21.15.177:3300/0 conn(0x7f59d0131b90 0x7f59d013a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f59ced9d640 10 monclient(hunting): get_auth_request con 0x7f59d0131b90 auth_method 0 2024-09-15T03:58:46.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f59ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f59ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f59ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6a180a24a0 0x7f6a1809ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6a180a24a0 0x7f6a1809ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.110:59884/0 (socket says 172.21.15.110:59884) 2024-09-15T03:58:46.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 1 -- 172.21.15.110:0/665125624 learned_addr learned my addr 172.21.15.110:0/665125624 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:46.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 10 monclient(hunting): get_auth_request con 0x7f6a180a24a0 auth_method 0 2024-09-15T03:58:46.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f59ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f59ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f59ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 10 monclient(hunting): handle_auth_done global_id 4261 payload 274 2024-09-15T03:58:46.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 1 -- 172.21.15.110:0/665125624 >> v1:172.21.15.177:6789/0 conn(0x7f6a18067ca0 legacy=0x7f6a18136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:46.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 1 -- 172.21.15.110:0/665125624 >> v1:172.21.15.110:6789/0 conn(0x7f6a18067130 legacy=0x7f6a18067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:46.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 10 monclient: _finish_hunting 0 2024-09-15T03:58:46.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 1 monclient: found mon.noname-c 2024-09-15T03:58:46.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:46.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:58:46.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.727+0000 7f6a1dab9640 1 -- 172.21.15.110:0/665125624 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a1809cb90 con 0x7f6a180a24a0 2024-09-15T03:58:46.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ce59c640 1 -- 172.21.15.110:0/2816115934 >> v1:172.21.15.110:6789/0 conn(0x7f59d00a1a10 legacy=0x7f59d009e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:46.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675387295 0 0) 0x7f59d009f2c0 con 0x7f59d00a1a10 2024-09-15T03:58:46.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient(hunting): my global_id is 4260 2024-09-15T03:58:46.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cdd9b640 1 -- 172.21.15.110:0/2816115934 >> v1:172.21.15.177:6789/0 conn(0x7f59d009ee80 legacy=0x7f59d01369e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:46.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b80034e0 con 0x7f59d00a1a10 2024-09-15T03:58:46.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910609494 0 0) 0x7f59d00a0850 con 0x7f59d009ee80 2024-09-15T03:58:46.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient(hunting): my global_id is 4264 2024-09-15T03:58:46.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d009f2c0 con 0x7f59d009ee80 2024-09-15T03:58:46.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1915715380 0 0) 0x7f59b80034e0 con 0x7f59d00a1a10 2024-09-15T03:58:46.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 >> v2:172.21.15.177:3300/0 conn(0x7f59d0131b90 msgr2=0x7f59d013a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:46.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 --2- 172.21.15.110:0/2816115934 >> v2:172.21.15.177:3300/0 conn(0x7f59d0131b90 0x7f59d013a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 >> v1:172.21.15.177:6789/0 conn(0x7f59d009ee80 legacy=0x7f59d01369e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient: _finish_hunting 0 2024-09-15T03:58:46.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 monclient: found mon.noname-a-legacy 2024-09-15T03:58:46.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:46.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.110:6789/0 2024-09-15T03:58:46.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d009cc80 con 0x7f59d00a1a10 2024-09-15T03:58:46.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient: _finish_auth 0 2024-09-15T03:58:46.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:46.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3165669920 0 0) 0x7f59c0001720 con 0x7f59d00a1a10 2024-09-15T03:58:46.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59d4c89640 5 monclient: authenticate success, global_id 4260 2024-09-15T03:58:46.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59d4c89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:46.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T03:58:46.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient: dump: 2024-09-15T03:58:46.745 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.745 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] but i'm connected to v1:172.21.15.110:6789/0, reconnecting 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 >> v1:172.21.15.110:6789/0 conn(0x7f59d00a1a10 legacy=0x7f59d009e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 --2- 172.21.15.110:0/2816115934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59b8003c30 0x7f59b8004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient(hunting): picked mon.c con 0x7f59b8003c30 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:46.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 --2- 172.21.15.110:0/2816115934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59b8004780 0x7f59b8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient(hunting): picked mon.b con 0x7f59b8004780 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 --2- 172.21.15.110:0/2816115934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59b80051f0 0x7f59b800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient(hunting): picked mon.a con 0x7f59b80051f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d009c9a0 con 0x7f59b80051f0 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b800c1a0 con 0x7f59b8004780 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b800c360 con 0x7f59b8003c30 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59cd59a640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59d4c89640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ce59c640 1 --2- 172.21.15.110:0/2816115934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59b8003c30 0x7f59b8004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ce59c640 10 monclient(hunting): get_auth_request con 0x7f59b8003c30 auth_method 0 2024-09-15T03:58:46.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ced9d640 1 --2- 172.21.15.110:0/2816115934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59b80051f0 0x7f59b800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ce59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:58:46.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ced9d640 10 monclient(hunting): get_auth_request con 0x7f59b80051f0 auth_method 0 2024-09-15T03:58:46.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ced9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:58:46.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:58:46.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:58:46.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ced9d640 10 monclient(hunting): handle_auth_done global_id 4260 payload 293 2024-09-15T03:58:46.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ced9d640 1 -- 172.21.15.110:0/2816115934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59b8003c30 msgr2=0x7f59b8004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:46.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ced9d640 1 --2- 172.21.15.110:0/2816115934 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59b8003c30 0x7f59b8004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ced9d640 1 -- 172.21.15.110:0/2816115934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59b8004780 msgr2=0x7f59b8004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:46.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ced9d640 1 --2- 172.21.15.110:0/2816115934 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59b8004780 0x7f59b8004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.729+0000 7f59ced9d640 10 monclient: _finish_hunting 0 2024-09-15T03:58:46.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f59ced9d640 1 monclient: found mon.a 2024-09-15T03:58:46.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f59ced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:46.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f59ced9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:46.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f59ced9d640 1 -- 172.21.15.110:0/2816115934 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b80034e0 con 0x7f59b80051f0 2024-09-15T03:58:46.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f59ced9d640 10 monclient: _finish_auth 0 2024-09-15T03:58:46.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f59ced9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:46.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1dab9640 1 -- 172.21.15.110:0/665125624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a180a24a0 msgr2=0x7f6a1809ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:46.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1dab9640 1 --2- 172.21.15.110:0/665125624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a180a24a0 0x7f6a1809ddb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6a14009910 tx=0x7f6a140091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6414ead9223a9ca1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:46.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f59ced9d640 1 -- 172.21.15.110:0/2816115934 reap_dead start 2024-09-15T03:58:46.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1cab7640 1 -- 172.21.15.110:0/665125624 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a14032440 con 0x7f6a180a24a0 2024-09-15T03:58:46.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1cab7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1cab7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1cab7640 10 monclient: dump: 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1cab7640 10 monclient: _finish_auth 0 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1cab7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:46.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1cab7640 1 -- 172.21.15.110:0/665125624 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a140325a0 con 0x7f6a180a24a0 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1fd1d640 5 monclient: authenticate success, global_id 4261 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1fd1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1cab7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1fd1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1cab7640 1 -- 172.21.15.110:0/665125624 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a14032940 con 0x7f6a180a24a0 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1fd1d640 4 set_mon_vals no callback set 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1cab7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1cab7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.730+0000 7f6a1cab7640 10 monclient: dump: 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.754 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.755 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.755 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.755 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.731+0000 7f6a1fd1d640 10 monclient: shutdown 2024-09-15T03:58:46.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.731+0000 7f6a1fd1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:46.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.731+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/665125624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a180a24a0 msgr2=0x7f6a1809ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.731+0000 7f6a1fd1d640 1 --2- 172.21.15.110:0/665125624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a180a24a0 0x7f6a1809ddb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6a14009910 tx=0x7f6a140091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.731+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/665125624 shutdown_connections 2024-09-15T03:58:46.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.731+0000 7f6a1fd1d640 1 --2- 172.21.15.110:0/665125624 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a180a24a0 0x7f6a1809ddb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.731+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/665125624 >> 172.21.15.110:0/665125624 conn(0x7f6a18068870 msgr2=0x7f6a180a1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:46.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.731+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/665125624 shutdown_connections 2024-09-15T03:58:46.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.731+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/665125624 wait complete. 2024-09-15T03:58:46.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.731+0000 7f6a1fd1d640 10 monclient: build_initial_monmap 2024-09-15T03:58:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.731+0000 7f6a1fd1d640 10 monclient: monmap: 2024-09-15T03:58:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:46.756 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 1 Processor -- start 2024-09-15T03:58:46.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 1 -- start start 2024-09-15T03:58:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 10 monclient: init 2024-09-15T03:58:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:58:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a18067130 0x7f6a18141640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 10 monclient(hunting): picked mon.noname-b con 0x7f6a18067130 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a18067ca0 0x7f6a18141b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 10 monclient(hunting): picked mon.noname-a con 0x7f6a18067ca0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a180a24a0 0x7f6a18135e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 10 monclient(hunting): picked mon.noname-c con 0x7f6a180a24a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1806e910 con 0x7f6a18067ca0 2024-09-15T03:58:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a18132380 con 0x7f6a18067130 2024-09-15T03:58:46.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1806bcb0 con 0x7f6a180a24a0 2024-09-15T03:58:46.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:46.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1fd1d640 10 monclient(hunting): authenticate will time out at 750.377620s 2024-09-15T03:58:46.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1e2ba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a180a24a0 0x7f6a18135e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1e2ba640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a180a24a0 0x7f6a18135e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55078/0 (socket says 172.21.15.110:55078) 2024-09-15T03:58:46.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1e2ba640 1 -- 172.21.15.110:0/1909243097 learned_addr learned my addr 172.21.15.110:0/1909243097 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:46.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1e2ba640 10 monclient(hunting): get_auth_request con 0x7f6a180a24a0 auth_method 0 2024-09-15T03:58:46.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f59ced9d640 1 -- 172.21.15.110:0/2816115934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59b80051f0 msgr2=0x7f59b800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:46.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f59ced9d640 1 --2- 172.21.15.110:0/2816115934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59b80051f0 0x7f59b800b9d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f59c800e210 tx=0x7f59c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6bef4e18a85fceb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:46.759 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:58:46.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1e2ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1e2ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1e2ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1dab9640 1 --2- 172.21.15.110:0/1909243097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a18067130 0x7f6a18141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1d2b8640 1 --2- 172.21.15.110:0/1909243097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a18067ca0 0x7f6a18141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1e2ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.760 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1e2ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1e2ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1dab9640 10 monclient(hunting): get_auth_request con 0x7f6a18067130 auth_method 0 2024-09-15T03:58:46.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1dab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1dab9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1dab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1d2b8640 10 monclient(hunting): get_auth_request con 0x7f6a18067ca0 auth_method 0 2024-09-15T03:58:46.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1d2b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1d2b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1d2b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1d2b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1d2b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1d2b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1dab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f6a1dab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c802c040 con 0x7f59b80051f0 2024-09-15T03:58:46.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f59cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f59cd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:46.762 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.732+0000 7f59cd59a640 10 monclient: dump: 2024-09-15T03:58:46.762 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.762 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.763 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.763 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.763 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.763 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.763 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.763 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.763 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.763 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f6a1dab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f6a1e2ba640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2024-09-15T03:58:46.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f6a1e2ba640 1 -- 172.21.15.110:0/1909243097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a18067130 msgr2=0x7f6a18141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c800f040 con 0x7f59b80051f0 2024-09-15T03:58:46.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59d4c89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:46.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f6a1e2ba640 1 --2- 172.21.15.110:0/1909243097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a18067130 0x7f6a18141640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f6a1e2ba640 1 -- 172.21.15.110:0/1909243097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a18067ca0 msgr2=0x7f6a18141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f6a1e2ba640 1 --2- 172.21.15.110:0/1909243097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a18067ca0 0x7f6a18141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f6a1e2ba640 10 monclient: _finish_hunting 0 2024-09-15T03:58:46.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f6a1e2ba640 1 monclient: found mon.noname-c 2024-09-15T03:58:46.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f6a1e2ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:46.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f6a1e2ba640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:46.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f6a1e2ba640 1 -- 172.21.15.110:0/1909243097 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a14009450 con 0x7f6a180a24a0 2024-09-15T03:58:46.766 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:58:46.767 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:58:46.767 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:58:46.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59cd59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:46.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59d4c89640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:46.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59d4c89640 4 set_mon_vals no callback set 2024-09-15T03:58:46.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59cd59a640 1 -- 172.21.15.110:0/2816115934 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c8030860 con 0x7f59b80051f0 2024-09-15T03:58:46.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59cd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:46.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59cd59a640 10 monclient: dump: 2024-09-15T03:58:46.769 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.769 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.769 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.770 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.770 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.770 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.770 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.770 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.770 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.770 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59d4c89640 10 monclient: shutdown 2024-09-15T03:58:46.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59d4c89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:46.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59d4c89640 1 -- 172.21.15.110:0/2816115934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59b80051f0 msgr2=0x7f59b800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59d4c89640 1 --2- 172.21.15.110:0/2816115934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59b80051f0 0x7f59b800b9d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f59c800e210 tx=0x7f59c800b040 comp rx=0 tx=0).stop 2024-09-15T03:58:46.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59d4c89640 1 -- 172.21.15.110:0/2816115934 shutdown_connections 2024-09-15T03:58:46.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59d4c89640 1 --2- 172.21.15.110:0/2816115934 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59b80051f0 0x7f59b800b9d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59d4c89640 1 -- 172.21.15.110:0/2816115934 >> 172.21.15.110:0/2816115934 conn(0x7f59d006e860 msgr2=0x7f59d006ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:46.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59d4c89640 1 -- 172.21.15.110:0/2816115934 shutdown_connections 2024-09-15T03:58:46.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.733+0000 7f59d4c89640 1 -- 172.21.15.110:0/2816115934 wait complete. 2024-09-15T03:58:46.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.734+0000 7f59d4c89640 10 monclient: build_initial_monmap 2024-09-15T03:58:46.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.734+0000 7f59d4c89640 10 monclient: monmap: 2024-09-15T03:58:46.771 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:46.771 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:46.771 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:46.771 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:46.771 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:46.771 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.771 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:46.771 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:46.771 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:46.772 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.734+0000 7f59d4c89640 1 Processor -- start 2024-09-15T03:58:46.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.734+0000 7f59d4c89640 1 -- start start 2024-09-15T03:58:46.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.734+0000 7f59d4c89640 10 monclient: init 2024-09-15T03:58:46.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:46.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:58:46.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59b80051f0 0x7f59d009a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 10 monclient(hunting): picked mon.noname-c con 0x7f59b80051f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:46.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59d0136050 0x7f59d0130760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 10 monclient(hunting): picked mon.noname-a con 0x7f59d0136050 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:46.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d0130ca0 0x7f59d011b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 10 monclient(hunting): picked mon.noname-b con 0x7f59d0130ca0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:46.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d011db80 con 0x7f59d0136050 2024-09-15T03:58:46.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d009b970 con 0x7f59d0130ca0 2024-09-15T03:58:46.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d0141a70 con 0x7f59b80051f0 2024-09-15T03:58:46.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:46.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59d4c89640 10 monclient(hunting): authenticate will time out at 750.380096s 2024-09-15T03:58:46.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59d0136050 0x7f59d0130760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59ce59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59b80051f0 0x7f59d009a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59d0136050 0x7f59d0130760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56592/0 (socket says 172.21.15.110:56592) 2024-09-15T03:58:46.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59ce59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59b80051f0 0x7f59d009a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55086/0 (socket says 172.21.15.110:55086) 2024-09-15T03:58:46.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 1 -- 172.21.15.110:0/695862435 learned_addr learned my addr 172.21.15.110:0/695862435 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:46.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 10 monclient(hunting): get_auth_request con 0x7f59d0136050 auth_method 0 2024-09-15T03:58:46.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59ce59c640 10 monclient(hunting): get_auth_request con 0x7f59b80051f0 auth_method 0 2024-09-15T03:58:46.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a1e2ba640 1 -- 172.21.15.110:0/1909243097 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a180a24a0 msgr2=0x7f6a18135e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:46.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a1e2ba640 1 --2- 172.21.15.110:0/1909243097 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a180a24a0 0x7f6a18135e20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6a1000c8c0 tx=0x7f6a10002640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:46.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 10 monclient(hunting): handle_auth_done global_id 4266 payload 824 2024-09-15T03:58:46.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 1 -- 172.21.15.110:0/695862435 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59b80051f0 msgr2=0x7f59d009a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 1 --2- 172.21.15.110:0/695862435 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59b80051f0 0x7f59d009a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 1 -- 172.21.15.110:0/695862435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d0130ca0 msgr2=0x7f59d011b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 1 --2- 172.21.15.110:0/695862435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d0130ca0 0x7f59d011b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 10 monclient: _finish_hunting 0 2024-09-15T03:58:46.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 1 monclient: found mon.noname-a 2024-09-15T03:58:46.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:46.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:46.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59cdd9b640 1 -- 172.21.15.110:0/695862435 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d011d8a0 con 0x7f59d0136050 2024-09-15T03:58:46.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f59ce59c640 1 --2- 172.21.15.110:0/695862435 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59b80051f0 0x7f59d009a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:46.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a0dffb640 1 -- 172.21.15.110:0/1909243097 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1000aa00 con 0x7f6a180a24a0 2024-09-15T03:58:46.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a0dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:46.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a0dffb640 10 monclient: dump: 2024-09-15T03:58:46.778 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.778 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.778 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a0dffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a0dffb640 1 -- 172.21.15.110:0/1909243097 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1000ab80 con 0x7f6a180a24a0 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a1fd1d640 5 monclient: authenticate success, global_id 4277 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a1fd1d640 10 monclient: _renew_subs 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a1fd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/1909243097 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a1806b800 con 0x7f6a180a24a0 2024-09-15T03:58:46.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a0dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a0dffb640 1 -- 172.21.15.110:0/1909243097 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a10012420 con 0x7f6a180a24a0 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a0dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a0dffb640 10 monclient: dump: 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a0ffff640 4 set_mon_vals no callback set 2024-09-15T03:58:46.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a1fd1d640 10 monclient: _renew_subs 2024-09-15T03:58:46.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a1fd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:46.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.735+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/1909243097 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a181420c0 con 0x7f6a180a24a0 2024-09-15T03:58:46.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.736+0000 7f6a0dffb640 1 -- 172.21.15.110:0/1909243097 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f6a10012850 con 0x7f6a180a24a0 2024-09-15T03:58:46.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.736+0000 7f6a1fd1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:46.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.736+0000 7f6a1fd1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:46.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.736+0000 7f6a1fd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:46.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.736+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/1909243097 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a180008d0 con 0x7f6a180a24a0 2024-09-15T03:58:46.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.736+0000 7f6a0dffb640 1 --2- 172.21.15.110:0/1909243097 >> v2:172.21.15.177:6812/28201 conn(0x7f69fc038750 0x7f69fc03ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.736+0000 7f6a0dffb640 1 -- 172.21.15.110:0/1909243097 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f6a1004ba30 con 0x7f6a180a24a0 2024-09-15T03:58:46.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f6a1dab9640 1 --2- 172.21.15.110:0/1909243097 >> v2:172.21.15.177:6812/28201 conn(0x7f69fc038750 0x7f69fc03ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f6a1dab9640 10 monclient: get_auth_request con 0x7f69fc038750 auth_method 0 2024-09-15T03:58:46.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f59cdd9b640 1 -- 172.21.15.110:0/695862435 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59d0136050 msgr2=0x7f59d0130760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:46.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f59cdd9b640 1 --2- 172.21.15.110:0/695862435 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59d0136050 0x7f59d0130760 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f59c400c7c0 tx=0x7f59c4001b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:46.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f59be7fc640 1 -- 172.21.15.110:0/695862435 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c4007510 con 0x7f59d0136050 2024-09-15T03:58:46.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f59be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f59be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:46.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f59be7fc640 10 monclient: dump: 2024-09-15T03:58:46.784 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.784 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.784 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.784 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.784 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.784 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.784 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.784 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.784 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.784 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f59be7fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:46.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f59be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:46.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f59be7fc640 1 -- 172.21.15.110:0/695862435 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c4005780 con 0x7f59d0136050 2024-09-15T03:58:46.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f59be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:46.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f59be7fc640 1 -- 172.21.15.110:0/695862435 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c4005b40 con 0x7f59d0136050 2024-09-15T03:58:46.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f59d4c89640 5 monclient: authenticate success, global_id 4266 2024-09-15T03:58:46.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f59d4c89640 10 monclient: _renew_subs 2024-09-15T03:58:46.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.738+0000 7f59d4c89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:46.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.738+0000 7f59d4c89640 1 -- 172.21.15.110:0/695862435 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d011de60 con 0x7f59d0136050 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.737+0000 7f59ccd99640 4 set_mon_vals no callback set 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.738+0000 7f59be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.738+0000 7f59be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.738+0000 7f59be7fc640 10 monclient: dump: 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.738+0000 7f59d4c89640 10 monclient: _renew_subs 2024-09-15T03:58:46.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.738+0000 7f59d4c89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:46.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.738+0000 7f59d4c89640 1 -- 172.21.15.110:0/695862435 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d011b9b0 con 0x7f59d0136050 2024-09-15T03:58:46.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.738+0000 7f59be7fc640 1 -- 172.21.15.110:0/695862435 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f59c401b400 con 0x7f59d0136050 2024-09-15T03:58:46.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.738+0000 7f59d4c89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:46.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.738+0000 7f59d4c89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:46.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.738+0000 7f59d4c89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:46.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.738+0000 7f59d4c89640 1 -- 172.21.15.110:0/695862435 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d00647e0 con 0x7f59d0136050 2024-09-15T03:58:46.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.739+0000 7f59be7fc640 1 --2- 172.21.15.110:0/695862435 >> v2:172.21.15.177:6812/28201 conn(0x7f59b0038750 0x7f59b003ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.739+0000 7f59be7fc640 1 -- 172.21.15.110:0/695862435 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f59c404b070 con 0x7f59d0136050 2024-09-15T03:58:46.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.739+0000 7f59ce59c640 1 --2- 172.21.15.110:0/695862435 >> v2:172.21.15.177:6812/28201 conn(0x7f59b0038750 0x7f59b003ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.739+0000 7f59ce59c640 10 monclient: get_auth_request con 0x7f59b0038750 auth_method 0 2024-09-15T03:58:46.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.739+0000 7f6a1dab9640 1 --2- 172.21.15.110:0/1909243097 >> v2:172.21.15.177:6812/28201 conn(0x7f69fc038750 0x7f69fc03ac00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6a18064e70 tx=0x7f6a14032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:46.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.741+0000 7f59ce59c640 1 --2- 172.21.15.110:0/695862435 >> v2:172.21.15.177:6812/28201 conn(0x7f59b0038750 0x7f59b003ac00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f59c0007fd0 tx=0x7f59c0007a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:46.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.745+0000 7f6a0dffb640 1 -- 172.21.15.110:0/1909243097 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6a1001b400 con 0x7f6a180a24a0 2024-09-15T03:58:46.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.745+0000 7f6a0dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:46.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.745+0000 7f6a0dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:46.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.748+0000 7f59be7fc640 1 -- 172.21.15.110:0/695862435 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f59c401a720 con 0x7f59d0136050 2024-09-15T03:58:46.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.748+0000 7f59be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:46.789 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.748+0000 7f59be7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:46.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.891+0000 7f25d059d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:46.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.891+0000 7f25d059d640 10 monclient: build_initial_monmap 2024-09-15T03:58:46.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.891+0000 7f25d059d640 10 monclient: monmap: 2024-09-15T03:58:46.894 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:46.894 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.894 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:46.893895+0000 2024-09-15T03:58:46.894 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:46.893895+0000 2024-09-15T03:58:46.895 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:46.895 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.895 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:46.895 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:46.895 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:46.895 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:46.895 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:46.895 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.892+0000 7f25d059d640 1 Processor -- start 2024-09-15T03:58:46.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.892+0000 7f25d059d640 1 -- start start 2024-09-15T03:58:46.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.892+0000 7f25d059d640 10 monclient: init 2024-09-15T03:58:46.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25d059d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:46.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25d059d640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-15T03:58:46.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25d059d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25c80a1a10 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:46.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25d059d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25c809ee80 0x7f25c8131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25d059d640 10 monclient(hunting): picked mon.noname-b con 0x7f25c809ee80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:46.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25d059d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f25c81369e0 0x7f25c8138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25d059d640 10 monclient(hunting): picked mon.noname-a con 0x7f25c81369e0 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:46.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25d059d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c809f290 con 0x7f25c80a1a10 2024-09-15T03:58:46.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25d059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25d059d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c80a0850 con 0x7f25c81369e0 2024-09-15T03:58:46.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25d059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25d059d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c809c9a0 con 0x7f25c809ee80 2024-09-15T03:58:46.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25d059d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:46.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25d059d640 10 monclient(hunting): authenticate will time out at 750.538563s 2024-09-15T03:58:46.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25ceb3a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f25c81369e0 0x7f25c8138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25ceb3a640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f25c81369e0 0x7f25c8138dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56596/0 (socket says 172.21.15.110:56596) 2024-09-15T03:58:46.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25ceb3a640 1 -- 172.21.15.110:0/4036044235 learned_addr learned my addr 172.21.15.110:0/4036044235 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:46.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25ceb3a640 10 monclient(hunting): get_auth_request con 0x7f25c81369e0 auth_method 0 2024-09-15T03:58:46.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25ceb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25ceb3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25ceb3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25ceb3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25ceb3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25ceb3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.893+0000 7f25cdb38640 1 --2- 172.21.15.110:0/4036044235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25c809ee80 0x7f25c8131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.894+0000 7f25ceb3a640 10 monclient(hunting): handle_auth_done global_id 4269 payload 274 2024-09-15T03:58:46.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.894+0000 7f25ceb3a640 1 -- 172.21.15.110:0/4036044235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25c809ee80 msgr2=0x7f25c8131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:46.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.894+0000 7f25ceb3a640 1 --2- 172.21.15.110:0/4036044235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25c809ee80 0x7f25c8131b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.894+0000 7f25ceb3a640 1 -- 172.21.15.110:0/4036044235 >> v1:172.21.15.177:6789/0 conn(0x7f25c80a1a10 legacy=0x7f25c809e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:46.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.894+0000 7f25ceb3a640 10 monclient: _finish_hunting 0 2024-09-15T03:58:46.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.894+0000 7f25ceb3a640 1 monclient: found mon.noname-a 2024-09-15T03:58:46.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.894+0000 7f25ceb3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:46.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.894+0000 7f25ceb3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:46.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.894+0000 7f25ceb3a640 1 -- 172.21.15.110:0/4036044235 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c809cc80 con 0x7f25c81369e0 2024-09-15T03:58:46.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.894+0000 7f25cdb38640 10 monclient: get_auth_request con 0x7f25c809ee80 auth_method 0 2024-09-15T03:58:46.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.894+0000 7f25cdb38640 1 --2- 172.21.15.110:0/4036044235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25c809ee80 0x7f25c8131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:58:46.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25ceb3a640 1 -- 172.21.15.110:0/4036044235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c81369e0 msgr2=0x7f25c8138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:46.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25ceb3a640 1 --2- 172.21.15.110:0/4036044235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c81369e0 0x7f25c8138dc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f25c0014ac0 tx=0x7f25c00143c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d6c763a21f3efe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:46.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25cd337640 1 -- 172.21.15.110:0/4036044235 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c00110d0 con 0x7f25c81369e0 2024-09-15T03:58:46.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25cd337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25cd337640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:46.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25cd337640 10 monclient: dump: 2024-09-15T03:58:46.907 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.907 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.907 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.907 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.907 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.907 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.907 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.907 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.908 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.908 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25cd337640 10 monclient: _finish_auth 0 2024-09-15T03:58:46.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25cd337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:46.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25cd337640 1 -- 172.21.15.110:0/4036044235 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c0011230 con 0x7f25c81369e0 2024-09-15T03:58:46.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25d059d640 5 monclient: authenticate success, global_id 4269 2024-09-15T03:58:46.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25cd337640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:46.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25cd337640 1 -- 172.21.15.110:0/4036044235 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c00115f0 con 0x7f25c81369e0 2024-09-15T03:58:46.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25d059d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:46.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25d059d640 4 set_mon_vals no callback set 2024-09-15T03:58:46.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25cd337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25cd337640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:46.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.897+0000 7f25cd337640 10 monclient: dump: 2024-09-15T03:58:46.909 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.909 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.909 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.909 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.909 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.909 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.909 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.909 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.909 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.909 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.898+0000 7f25d059d640 10 monclient: shutdown 2024-09-15T03:58:46.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.898+0000 7f25d059d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:46.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.898+0000 7f25d059d640 1 -- 172.21.15.110:0/4036044235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c81369e0 msgr2=0x7f25c8138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.898+0000 7f25d059d640 1 --2- 172.21.15.110:0/4036044235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c81369e0 0x7f25c8138dc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f25c0014ac0 tx=0x7f25c00143c0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.898+0000 7f25d059d640 1 -- 172.21.15.110:0/4036044235 shutdown_connections 2024-09-15T03:58:46.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.898+0000 7f25d059d640 1 --2- 172.21.15.110:0/4036044235 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c81369e0 0x7f25c8138dc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.898+0000 7f25d059d640 1 --2- 172.21.15.110:0/4036044235 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25c809ee80 0x7f25c8131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.898+0000 7f25d059d640 1 -- 172.21.15.110:0/4036044235 >> 172.21.15.110:0/4036044235 conn(0x7f25c806e860 msgr2=0x7f25c806ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:46.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.899+0000 7f25d059d640 1 -- 172.21.15.110:0/4036044235 shutdown_connections 2024-09-15T03:58:46.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.899+0000 7f25d059d640 1 -- 172.21.15.110:0/4036044235 wait complete. 2024-09-15T03:58:46.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.899+0000 7f25d059d640 10 monclient: build_initial_monmap 2024-09-15T03:58:46.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.899+0000 7f25d059d640 10 monclient: monmap: 2024-09-15T03:58:46.910 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:46.910 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:46.910 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:46.910 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:46.911 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:46.911 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.911 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:46.911 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:46.911 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:46.911 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 1 Processor -- start 2024-09-15T03:58:46.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 1 -- start start 2024-09-15T03:58:46.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 10 monclient: init 2024-09-15T03:58:46.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:46.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:58:46.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25c809ee80 0x7f25c8135f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 10 monclient(hunting): picked mon.noname-c con 0x7f25c809ee80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:46.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c80a1a10 0x7f25c8095a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 10 monclient(hunting): picked mon.noname-a con 0x7f25c80a1a10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:46.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25c81369e0 0x7f25c8095fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 10 monclient(hunting): picked mon.noname-b con 0x7f25c81369e0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:46.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80a4a20 con 0x7f25c80a1a10 2024-09-15T03:58:46.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80a11c0 con 0x7f25c81369e0 2024-09-15T03:58:46.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c8098a10 con 0x7f25c809ee80 2024-09-15T03:58:46.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:46.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25d059d640 10 monclient(hunting): authenticate will time out at 750.545768s 2024-09-15T03:58:46.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25cdb38640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c80a1a10 0x7f25c8095a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.900+0000 7f25ceb3a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25c81369e0 0x7f25c8095fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25c81369e0 0x7f25c8095fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.110:59920/0 (socket says 172.21.15.110:59920) 2024-09-15T03:58:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25cdb38640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c80a1a10 0x7f25c8095a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56610/0 (socket says 172.21.15.110:56610) 2024-09-15T03:58:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 1 -- 172.21.15.110:0/1150490368 learned_addr learned my addr 172.21.15.110:0/1150490368 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 10 monclient(hunting): get_auth_request con 0x7f25c81369e0 auth_method 0 2024-09-15T03:58:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ce339640 1 --2- 172.21.15.110:0/1150490368 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25c809ee80 0x7f25c8135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25cdb38640 10 monclient(hunting): get_auth_request con 0x7f25c80a1a10 auth_method 0 2024-09-15T03:58:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25cdb38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25cdb38640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25cdb38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ce339640 10 monclient(hunting): get_auth_request con 0x7f25c809ee80 auth_method 0 2024-09-15T03:58:46.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ce339640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ce339640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ce339640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25cdb38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25cdb38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25cdb38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ce339640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ce339640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ce339640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 10 monclient(hunting): handle_auth_done global_id 4270 payload 824 2024-09-15T03:58:46.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 1 -- 172.21.15.110:0/1150490368 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25c809ee80 msgr2=0x7f25c8135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 1 --2- 172.21.15.110:0/1150490368 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25c809ee80 0x7f25c8135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 1 -- 172.21.15.110:0/1150490368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c80a1a10 msgr2=0x7f25c8095a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 1 --2- 172.21.15.110:0/1150490368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c80a1a10 0x7f25c8095a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 10 monclient: _finish_hunting 0 2024-09-15T03:58:46.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 1 monclient: found mon.noname-b 2024-09-15T03:58:46.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:46.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-09-15T03:58:46.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ceb3a640 1 -- 172.21.15.110:0/1150490368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c0014600 con 0x7f25c81369e0 2024-09-15T03:58:46.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25cdb38640 1 --2- 172.21.15.110:0/1150490368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c80a1a10 0x7f25c8095a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:46.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.901+0000 7f25ce339640 1 --2- 172.21.15.110:0/1150490368 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25c809ee80 0x7f25c8135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:46.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.904+0000 7f25ceb3a640 1 -- 172.21.15.110:0/1150490368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25c81369e0 msgr2=0x7f25c8095fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:46.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.904+0000 7f25ceb3a640 1 --2- 172.21.15.110:0/1150490368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25c81369e0 0x7f25c8095fb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f25c0014b20 tx=0x7f25c003d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:46.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.904+0000 7f25be7fc640 1 -- 172.21.15.110:0/1150490368 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0011a60 con 0x7f25c81369e0 2024-09-15T03:58:46.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.904+0000 7f25be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.904+0000 7f25be7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:46.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.904+0000 7f25be7fc640 10 monclient: dump: 2024-09-15T03:58:46.919 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.919 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.919 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.919 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.919 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.919 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.919 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.919 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.919 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.919 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.904+0000 7f25be7fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:46.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.904+0000 7f25be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:46.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.904+0000 7f25be7fc640 1 -- 172.21.15.110:0/1150490368 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c003bcc0 con 0x7f25c81369e0 2024-09-15T03:58:46.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.904+0000 7f25be7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:46.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.904+0000 7f25ccb36640 4 set_mon_vals no callback set 2024-09-15T03:58:46.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.904+0000 7f25be7fc640 1 -- 172.21.15.110:0/1150490368 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0046680 con 0x7f25c81369e0 2024-09-15T03:58:46.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.904+0000 7f25be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.904+0000 7f25be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:46.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.905+0000 7f25be7fc640 10 monclient: dump: 2024-09-15T03:58:46.920 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.920 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.920 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.920 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.920 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.920 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.920 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.921 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.921 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.921 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.905+0000 7f25d059d640 5 monclient: authenticate success, global_id 4270 2024-09-15T03:58:46.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.905+0000 7f25d059d640 10 monclient: _renew_subs 2024-09-15T03:58:46.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.905+0000 7f25d059d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:46.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.905+0000 7f25d059d640 1 -- 172.21.15.110:0/1150490368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c8098730 con 0x7f25c81369e0 2024-09-15T03:58:46.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.905+0000 7f25d059d640 10 monclient: _renew_subs 2024-09-15T03:58:46.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.905+0000 7f25d059d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:46.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.905+0000 7f25d059d640 1 -- 172.21.15.110:0/1150490368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c8097840 con 0x7f25c81369e0 2024-09-15T03:58:46.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.906+0000 7f25d059d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:46.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.906+0000 7f25d059d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:46.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.906+0000 7f25d059d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:46.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.906+0000 7f25d059d640 1 -- 172.21.15.110:0/1150490368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c8096cf0 con 0x7f25c81369e0 2024-09-15T03:58:46.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.906+0000 7f25be7fc640 1 -- 172.21.15.110:0/1150490368 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f25c0003040 con 0x7f25c81369e0 2024-09-15T03:58:46.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.906+0000 7f25be7fc640 1 --2- 172.21.15.110:0/1150490368 >> v2:172.21.15.177:6812/28201 conn(0x7f25ac038750 0x7f25ac03ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.906+0000 7f25be7fc640 1 -- 172.21.15.110:0/1150490368 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f25c00754c0 con 0x7f25c81369e0 2024-09-15T03:58:46.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.907+0000 7f25ce339640 1 --2- 172.21.15.110:0/1150490368 >> v2:172.21.15.177:6812/28201 conn(0x7f25ac038750 0x7f25ac03ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.907+0000 7f25ce339640 10 monclient: get_auth_request con 0x7f25ac038750 auth_method 0 2024-09-15T03:58:46.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.909+0000 7f25ce339640 1 --2- 172.21.15.110:0/1150490368 >> v2:172.21.15.177:6812/28201 conn(0x7f25ac038750 0x7f25ac03ac00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f25c4007fd0 tx=0x7f25c4007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:46.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.911+0000 7f25be7fc640 1 -- 172.21.15.110:0/1150490368 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f25c0039ae0 con 0x7f25c81369e0 2024-09-15T03:58:46.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.911+0000 7f25be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:46.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.911+0000 7f25be7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:46.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.924+0000 7f6a1fd1d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:58:46.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.924+0000 7f6a1fd1d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:58:46.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.924+0000 7f6a1fd1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:46.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.924+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/1909243097 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6a180647e0 con 0x7f6a180a24a0 2024-09-15T03:58:46.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.925+0000 7f6a0dffb640 1 -- 172.21.15.110:0/1909243097 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6a10016070 con 0x7f6a180a24a0 2024-09-15T03:58:46.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.925+0000 7f6a0dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:58:46.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.925+0000 7f6a0dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:46.927 INFO:teuthology.orchestra.run.smithi110.stdout:47244640258 2024-09-15T03:58:46.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.927+0000 7f4bbac20640 10 monclient: get_monmap_and_config 2024-09-15T03:58:46.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.927+0000 7f4bbac20640 10 monclient: build_initial_monmap 2024-09-15T03:58:46.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.927+0000 7f4bbac20640 10 monclient: monmap: 2024-09-15T03:58:46.929 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:46.929 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.929 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:46.929197+0000 2024-09-15T03:58:46.929 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:46.929197+0000 2024-09-15T03:58:46.929 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:46.929 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.929 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:46.929 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:46.929 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:46.929 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:46.929 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:46.929 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.927+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/1909243097 >> v2:172.21.15.177:6812/28201 conn(0x7f69fc038750 msgr2=0x7f69fc03ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.927+0000 7f6a1fd1d640 1 --2- 172.21.15.110:0/1909243097 >> v2:172.21.15.177:6812/28201 conn(0x7f69fc038750 0x7f69fc03ac00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6a18064e70 tx=0x7f6a14032000 comp rx=0 tx=0).stop 2024-09-15T03:58:46.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.927+0000 7f6a1fd1d640 10 monclient: shutdown 2024-09-15T03:58:46.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.927+0000 7f6a1fd1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:46.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.927+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/1909243097 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a180a24a0 msgr2=0x7f6a18135e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.927+0000 7f6a1fd1d640 1 --2- 172.21.15.110:0/1909243097 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a180a24a0 0x7f6a18135e20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6a1000c8c0 tx=0x7f6a10002640 comp rx=0 tx=0).stop 2024-09-15T03:58:46.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.928+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/1909243097 shutdown_connections 2024-09-15T03:58:46.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.928+0000 7f4bbac20640 1 Processor -- start 2024-09-15T03:58:46.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.928+0000 7f6a1fd1d640 1 --2- 172.21.15.110:0/1909243097 >> v2:172.21.15.177:6812/28201 conn(0x7f69fc038750 0x7f69fc03ac00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.928+0000 7f6a1fd1d640 1 --2- 172.21.15.110:0/1909243097 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6a180a24a0 0x7f6a18135e20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.928+0000 7f4bbac20640 1 -- start start 2024-09-15T03:58:46.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.928+0000 7f4bbac20640 10 monclient: init 2024-09-15T03:58:46.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.928+0000 7f6a1fd1d640 1 --2- 172.21.15.110:0/1909243097 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6a18067ca0 0x7f6a18141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.928+0000 7f6a1fd1d640 1 --2- 172.21.15.110:0/1909243097 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6a18067130 0x7f6a18141640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.928+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/1909243097 >> 172.21.15.110:0/1909243097 conn(0x7f6a18068870 msgr2=0x7f6a180961e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:46.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.928+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/1909243097 shutdown_connections 2024-09-15T03:58:46.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.928+0000 7f6a1fd1d640 1 -- 172.21.15.110:0/1909243097 wait complete. 2024-09-15T03:58:46.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bbac20640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:46.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bbac20640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T03:58:46.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bbac20640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4bb40a1a10 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:46.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bbac20640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4bb409ee80 0x7f4bb4131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bbac20640 10 monclient(hunting): picked mon.noname-c con 0x7f4bb409ee80 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:46.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bbac20640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4bb41369e0 0x7f4bb4138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bbac20640 10 monclient(hunting): picked mon.noname-a con 0x7f4bb41369e0 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:46.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bb91bd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4bb41369e0 0x7f4bb4138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bb3fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4bb409ee80 0x7f4bb4131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bb3fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4bb409ee80 0x7f4bb4131b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.110:59934/0 (socket says 172.21.15.110:59934) 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bb3fff640 1 -- 172.21.15.110:0/384651312 learned_addr learned my addr 172.21.15.110:0/384651312 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bbac20640 1 -- 172.21.15.110:0/384651312 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb409f2c0 con 0x7f4bb40a1a10 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bbac20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bbac20640 1 -- 172.21.15.110:0/384651312 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bb40a0850 con 0x7f4bb41369e0 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bbac20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.929+0000 7f4bbac20640 1 -- 172.21.15.110:0/384651312 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bb409c9a0 con 0x7f4bb409ee80 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bbac20640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bbac20640 10 monclient(hunting): authenticate will time out at 750.575078s 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 10 monclient(hunting): get_auth_request con 0x7f4bb409ee80 auth_method 0 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb91bd640 10 monclient(hunting): get_auth_request con 0x7f4bb41369e0 auth_method 0 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb91bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb91bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb91bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb91bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb91bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb91bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 10 monclient(hunting): handle_auth_done global_id 4273 payload 274 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 1 -- 172.21.15.110:0/384651312 >> v2:172.21.15.110:3300/0 conn(0x7f4bb41369e0 msgr2=0x7f4bb4138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 1 --2- 172.21.15.110:0/384651312 >> v2:172.21.15.110:3300/0 conn(0x7f4bb41369e0 0x7f4bb4138dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 1 -- 172.21.15.110:0/384651312 >> v1:172.21.15.177:6789/0 conn(0x7f4bb40a1a10 legacy=0x7f4bb409e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:46.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 1 monclient: found mon.noname-c 2024-09-15T03:58:46.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:46.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:58:46.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb3fff640 1 -- 172.21.15.110:0/384651312 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb409cc80 con 0x7f4bb409ee80 2024-09-15T03:58:46.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.930+0000 7f4bb91bd640 1 --2- 172.21.15.110:0/384651312 >> v2:172.21.15.110:3300/0 conn(0x7f4bb41369e0 0x7f4bb4138dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:46.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.933+0000 7f10cf19d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:46.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.933+0000 7f10cf19d640 10 monclient: build_initial_monmap 2024-09-15T03:58:46.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.933+0000 7f10cf19d640 10 monclient: monmap: 2024-09-15T03:58:46.941 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:46.941 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.941 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:46.935195+0000 2024-09-15T03:58:46.941 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:46.935195+0000 2024-09-15T03:58:46.941 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:46.941 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.941 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:46.941 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:46.941 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:46.941 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:46.941 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:46.941 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.933+0000 7f4bb3fff640 1 -- 172.21.15.110:0/384651312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb409ee80 msgr2=0x7f4bb4131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:46.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bb3fff640 1 --2- 172.21.15.110:0/384651312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb409ee80 0x7f4bb4131b90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4bb40a2790 tx=0x7f4ba40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5b83ba62dcaedc1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:46.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f10cf19d640 1 Processor -- start 2024-09-15T03:58:46.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f10cf19d640 1 -- start start 2024-09-15T03:58:46.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f10cf19d640 10 monclient: init 2024-09-15T03:58:46.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bb37fe640 1 -- 172.21.15.110:0/384651312 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4032440 con 0x7f4bb409ee80 2024-09-15T03:58:46.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bb37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:46.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bb37fe640 10 monclient: dump: 2024-09-15T03:58:46.945 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.945 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.945 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.945 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.945 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.945 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.945 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.945 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.945 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.945 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bb37fe640 10 monclient: _finish_auth 0 2024-09-15T03:58:46.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:46.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bb37fe640 1 -- 172.21.15.110:0/384651312 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba40325a0 con 0x7f4bb409ee80 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bb37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bb37fe640 1 -- 172.21.15.110:0/384651312 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4032940 con 0x7f4bb409ee80 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bb37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bb37fe640 10 monclient: dump: 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.946 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bbac20640 5 monclient: authenticate success, global_id 4273 2024-09-15T03:58:46.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bbac20640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:46.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.934+0000 7f4bbac20640 4 set_mon_vals no callback set 2024-09-15T03:58:46.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:46.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-15T03:58:46.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10c80a1a10 0x7f10c8131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 10 monclient(hunting): picked mon.noname-b con 0x7f10c80a1a10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:46.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f10c809e770 0x7f10c809eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 10 monclient(hunting): picked mon.noname-c con 0x7f10c809e770 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:46.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f10c809f800 0x7f10c809fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 10 monclient(hunting): picked mon.noname-a con 0x7f10c809f800 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:46.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c80a0850 con 0x7f10c809f800 2024-09-15T03:58:46.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f4bbac20640 10 monclient: shutdown 2024-09-15T03:58:46.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10c7fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f10c809e770 0x7f10c809eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10c7fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f10c809e770 0x7f10c809eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.110:59936/0 (socket says 172.21.15.110:59936) 2024-09-15T03:58:46.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10c7fff640 1 -- 172.21.15.110:0/575646499 learned_addr learned my addr 172.21.15.110:0/575646499 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:46.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cd73a640 1 --2- 172.21.15.110:0/575646499 >> v2:172.21.15.110:3300/0 conn(0x7f10c809f800 0x7f10c809fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 1 -- 172.21.15.110:0/575646499 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c809c9a0 con 0x7f10c809e770 2024-09-15T03:58:46.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 1 -- 172.21.15.110:0/575646499 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c809cc80 con 0x7f10c80a1a10 2024-09-15T03:58:46.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f4bbac20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:46.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:46.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cf19d640 10 monclient(hunting): authenticate will time out at 750.580683s 2024-09-15T03:58:46.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10c7fff640 10 monclient(hunting): get_auth_request con 0x7f10c809e770 auth_method 0 2024-09-15T03:58:46.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cd73a640 10 monclient(hunting): get_auth_request con 0x7f10c809f800 auth_method 0 2024-09-15T03:58:46.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cd73a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cd73a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cd73a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10ccf39640 1 --2- 172.21.15.110:0/575646499 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10c80a1a10 0x7f10c8131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f4bbac20640 1 -- 172.21.15.110:0/384651312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb409ee80 msgr2=0x7f4bb4131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10ccf39640 10 monclient(hunting): get_auth_request con 0x7f10c80a1a10 auth_method 0 2024-09-15T03:58:46.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10ccf39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10ccf39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10ccf39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f4bbac20640 1 --2- 172.21.15.110:0/384651312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb409ee80 0x7f4bb4131b90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4bb40a2790 tx=0x7f4ba40091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cd73a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cd73a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10cd73a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10ccf39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10ccf39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f10ccf39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f4bbac20640 1 -- 172.21.15.110:0/384651312 shutdown_connections 2024-09-15T03:58:46.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f4bbac20640 1 --2- 172.21.15.110:0/384651312 >> v2:172.21.15.110:3300/0 conn(0x7f4bb41369e0 0x7f4bb4138dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.936+0000 7f10c7fff640 10 monclient(hunting): handle_auth_done global_id 4276 payload 274 2024-09-15T03:58:46.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.936+0000 7f10c7fff640 1 -- 172.21.15.110:0/575646499 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10c80a1a10 msgr2=0x7f10c8131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:46.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f4bbac20640 1 --2- 172.21.15.110:0/384651312 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb409ee80 0x7f4bb4131b90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.935+0000 7f4bbac20640 1 -- 172.21.15.110:0/384651312 >> 172.21.15.110:0/384651312 conn(0x7f4bb406e860 msgr2=0x7f4bb406ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:46.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.936+0000 7f4bbac20640 1 -- 172.21.15.110:0/384651312 shutdown_connections 2024-09-15T03:58:46.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.936+0000 7f10c7fff640 1 --2- 172.21.15.110:0/575646499 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10c80a1a10 0x7f10c8131b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.936+0000 7f10c7fff640 1 -- 172.21.15.110:0/575646499 >> v2:172.21.15.110:3300/0 conn(0x7f10c809f800 msgr2=0x7f10c809fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:46.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.936+0000 7f10c7fff640 1 --2- 172.21.15.110:0/575646499 >> v2:172.21.15.110:3300/0 conn(0x7f10c809f800 0x7f10c809fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.936+0000 7f10c7fff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:46.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.936+0000 7f10c7fff640 1 monclient: found mon.noname-c 2024-09-15T03:58:46.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.936+0000 7f10c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:46.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.936+0000 7f10c7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:58:46.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.936+0000 7f10c7fff640 1 -- 172.21.15.110:0/575646499 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c809c3f0 con 0x7f10c809e770 2024-09-15T03:58:46.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.936+0000 7f10cd73a640 1 --2- 172.21.15.110:0/575646499 >> v2:172.21.15.110:3300/0 conn(0x7f10c809f800 0x7f10c809fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:46.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.936+0000 7f10ccf39640 1 --2- 172.21.15.110:0/575646499 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10c80a1a10 0x7f10c8131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.936+0000 7f4bbac20640 1 -- 172.21.15.110:0/384651312 wait complete. 2024-09-15T03:58:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.937+0000 7f4bbac20640 10 monclient: build_initial_monmap 2024-09-15T03:58:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.937+0000 7f4bbac20640 10 monclient: monmap: 2024-09-15T03:58:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:46.961 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:46.962 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.937+0000 7f4bbac20640 1 Processor -- start 2024-09-15T03:58:46.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.937+0000 7f4bbac20640 1 -- start start 2024-09-15T03:58:46.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.937+0000 7f4bbac20640 10 monclient: init 2024-09-15T03:58:46.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:46.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:58:46.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb409ee80 0x7f4bb409b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 10 monclient(hunting): picked mon.noname-b con 0x7f4bb409ee80 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:46.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bb40a1a10 0x7f4bb40971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 10 monclient(hunting): picked mon.noname-a con 0x7f4bb40a1a10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:46.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bb41369e0 0x7f4bb4097720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 10 monclient(hunting): picked mon.noname-c con 0x7f4bb41369e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:46.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb409b9c0 con 0x7f4bb40a1a10 2024-09-15T03:58:46.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb40a11c0 con 0x7f4bb409ee80 2024-09-15T03:58:46.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb40a0320 con 0x7f4bb41369e0 2024-09-15T03:58:46.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:46.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bbac20640 10 monclient(hunting): authenticate will time out at 750.583276s 2024-09-15T03:58:46.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bb40a1a10 0x7f4bb40971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bb40a1a10 0x7f4bb40971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56644/0 (socket says 172.21.15.110:56644) 2024-09-15T03:58:46.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 1 -- 172.21.15.110:0/1145755548 learned_addr learned my addr 172.21.15.110:0/1145755548 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:46.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 10 monclient(hunting): get_auth_request con 0x7f4bb40a1a10 auth_method 0 2024-09-15T03:58:46.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb91bd640 1 --2- 172.21.15.110:0/1145755548 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bb41369e0 0x7f4bb4097720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb91bd640 10 monclient(hunting): get_auth_request con 0x7f4bb41369e0 auth_method 0 2024-09-15T03:58:46.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb91bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb91bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb91bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb89bc640 1 --2- 172.21.15.110:0/1145755548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb409ee80 0x7f4bb409b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb89bc640 10 monclient(hunting): get_auth_request con 0x7f4bb409ee80 auth_method 0 2024-09-15T03:58:46.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb89bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb89bc640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb89bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.939+0000 7f10c7fff640 1 -- 172.21.15.110:0/575646499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10c809e770 msgr2=0x7f10c809eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:46.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.939+0000 7f10c7fff640 1 --2- 172.21.15.110:0/575646499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10c809e770 0x7f10c809eb80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f10c8069a30 tx=0x7f10b40091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bacc528c8d508ee9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:46.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2024-09-15T03:58:46.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 1 -- 172.21.15.110:0/1145755548 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bb41369e0 msgr2=0x7f4bb4097720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 1 --2- 172.21.15.110:0/1145755548 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bb41369e0 0x7f4bb4097720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 1 -- 172.21.15.110:0/1145755548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb409ee80 msgr2=0x7f4bb409b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 1 --2- 172.21.15.110:0/1145755548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb409ee80 0x7f4bb409b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:46.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.939+0000 7f10c77fe640 1 -- 172.21.15.110:0/575646499 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b4032440 con 0x7f10c809e770 2024-09-15T03:58:46.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.939+0000 7f10c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.939+0000 7f10c77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:46.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.939+0000 7f10c77fe640 10 monclient: dump: 2024-09-15T03:58:46.967 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.967 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.967 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.968 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.968 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.968 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.968 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.968 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.968 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.968 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.939+0000 7f10c77fe640 10 monclient: _finish_auth 0 2024-09-15T03:58:46.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.939+0000 7f10c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:46.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.939+0000 7f10c77fe640 1 -- 172.21.15.110:0/575646499 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b40325a0 con 0x7f10c809e770 2024-09-15T03:58:46.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.939+0000 7f10c77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:46.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.939+0000 7f10c77fe640 1 -- 172.21.15.110:0/575646499 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b4032940 con 0x7f10c809e770 2024-09-15T03:58:46.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.939+0000 7f10c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.939+0000 7f10c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:46.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.939+0000 7f10c77fe640 10 monclient: dump: 2024-09-15T03:58:46.969 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.969 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.969 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.969 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.969 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.969 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.969 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.969 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.969 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.969 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.940+0000 7f10cf19d640 5 monclient: authenticate success, global_id 4276 2024-09-15T03:58:46.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.940+0000 7f10cf19d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:46.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.940+0000 7f10cf19d640 4 set_mon_vals no callback set 2024-09-15T03:58:46.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 1 monclient: found mon.noname-a 2024-09-15T03:58:46.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:46.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:46.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.938+0000 7f4bb3fff640 1 -- 172.21.15.110:0/1145755548 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac009a20 con 0x7f4bb40a1a10 2024-09-15T03:58:46.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.940+0000 7f10cf19d640 10 monclient: shutdown 2024-09-15T03:58:46.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.940+0000 7f10cf19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:46.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.940+0000 7f10cf19d640 1 -- 172.21.15.110:0/575646499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10c809e770 msgr2=0x7f10c809eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:46.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.940+0000 7f10cf19d640 1 --2- 172.21.15.110:0/575646499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10c809e770 0x7f10c809eb80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f10c8069a30 tx=0x7f10b40091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.940+0000 7f10cf19d640 1 -- 172.21.15.110:0/575646499 shutdown_connections 2024-09-15T03:58:46.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.940+0000 7f10cf19d640 1 --2- 172.21.15.110:0/575646499 >> v2:172.21.15.110:3300/0 conn(0x7f10c809f800 0x7f10c809fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.940+0000 7f10cf19d640 1 --2- 172.21.15.110:0/575646499 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10c809e770 0x7f10c809eb80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.940+0000 7f10cf19d640 1 --2- 172.21.15.110:0/575646499 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10c80a1a10 0x7f10c8131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.940+0000 7f10cf19d640 1 -- 172.21.15.110:0/575646499 >> 172.21.15.110:0/575646499 conn(0x7f10c806e860 msgr2=0x7f10c806ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:46.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb3fff640 1 -- 172.21.15.110:0/1145755548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bb40a1a10 msgr2=0x7f4bb40971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:46.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb3fff640 1 --2- 172.21.15.110:0/1145755548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bb40a1a10 0x7f4bb40971e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009940 tx=0x7f4ba4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:46.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f10cf19d640 1 -- 172.21.15.110:0/575646499 shutdown_connections 2024-09-15T03:58:46.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f10cf19d640 1 -- 172.21.15.110:0/575646499 wait complete. 2024-09-15T03:58:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f5e0dc68640 10 monclient: get_monmap_and_config 2024-09-15T03:58:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f5e0dc68640 10 monclient: build_initial_monmap 2024-09-15T03:58:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb0ff9640 1 -- 172.21.15.110:0/1145755548 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4031940 con 0x7f4bb40a1a10 2024-09-15T03:58:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f5e0dc68640 10 monclient: monmap: 2024-09-15T03:58:46.973 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:46.974 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.974 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:46.943379+0000 2024-09-15T03:58:46.974 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:46.943379+0000 2024-09-15T03:58:46.974 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:46.974 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.974 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:46.974 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:46.974 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:46.974 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:46.974 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:46.974 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:46.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb0ff9640 10 monclient: dump: 2024-09-15T03:58:46.975 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.976 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.976 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.976 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.976 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.976 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.976 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.976 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.976 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.976 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb0ff9640 10 monclient: _finish_auth 0 2024-09-15T03:58:46.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:46.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb0ff9640 1 -- 172.21.15.110:0/1145755548 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba4031ac0 con 0x7f4bb40a1a10 2024-09-15T03:58:46.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb0ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:46.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb0ff9640 1 -- 172.21.15.110:0/1145755548 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4031e80 con 0x7f4bb40a1a10 2024-09-15T03:58:46.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:46.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f10cf19d640 10 monclient: build_initial_monmap 2024-09-15T03:58:46.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:46.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f10cf19d640 10 monclient: monmap: 2024-09-15T03:58:46.978 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:46.978 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:46.978 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:46.978 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:46.978 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:46.978 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.978 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:46.978 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:46.979 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:46.979 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb0ff9640 10 monclient: dump: 2024-09-15T03:58:46.980 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:46.980 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:46.980 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.981 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:46.981 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:46.981 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:46.981 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:46.981 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:46.981 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:46.981 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:46.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.941+0000 7f4bb2ffd640 4 set_mon_vals no callback set 2024-09-15T03:58:46.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.942+0000 7f4bbac20640 5 monclient: authenticate success, global_id 4281 2024-09-15T03:58:46.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.942+0000 7f4bbac20640 10 monclient: _renew_subs 2024-09-15T03:58:46.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.942+0000 7f4bbac20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:46.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.942+0000 7f4bbac20640 1 -- 172.21.15.110:0/1145755548 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba4009450 con 0x7f4bb40a1a10 2024-09-15T03:58:46.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.942+0000 7f5e0dc68640 1 Processor -- start 2024-09-15T03:58:46.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.942+0000 7f4bbac20640 10 monclient: _renew_subs 2024-09-15T03:58:46.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.942+0000 7f5e0dc68640 1 -- start start 2024-09-15T03:58:46.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.942+0000 7f5e0dc68640 10 monclient: init 2024-09-15T03:58:46.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.942+0000 7f4bbac20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:46.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.942+0000 7f4bbac20640 1 -- 172.21.15.110:0/1145755548 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb4098c90 con 0x7f4bb40a1a10 2024-09-15T03:58:46.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.942+0000 7f10cf19d640 1 Processor -- start 2024-09-15T03:58:46.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.942+0000 7f10cf19d640 1 -- start start 2024-09-15T03:58:46.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.942+0000 7f10cf19d640 10 monclient: init 2024-09-15T03:58:46.984 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.5 2024-09-15T03:58:46.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f10cf19d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:46.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f10cf19d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:58:46.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f10cf19d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10c809e770 0x7f10c806c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f10cf19d640 10 monclient(hunting): picked mon.noname-a con 0x7f10c809e770 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:46.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e0dc68640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:46.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e0dc68640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-15T03:58:46.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f10ccf39640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10c809e770 0x7f10c806c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f10ccf39640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10c809e770 0x7f10c806c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56660/0 (socket says 172.21.15.110:56660) 2024-09-15T03:58:46.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e0dc68640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e080a1a10 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:46.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e0dc68640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5e0809ee80 0x7f5e08131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e0dc68640 10 monclient(hunting): picked mon.noname-c con 0x7f5e0809ee80 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:46.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e0dc68640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e081369e0 0x7f5e08138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e0dc68640 10 monclient(hunting): picked mon.noname-b con 0x7f5e081369e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:46.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e077fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f5e080a1a10 0x7f5e0809e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33846/0 (socket says 172.21.15.110:33846) 2024-09-15T03:58:46.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e077fe640 1 -- 172.21.15.110:0/3523483106 learned_addr learned my addr 172.21.15.110:0/3523483106 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:46.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e0dc68640 1 -- 172.21.15.110:0/3523483106 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0809f2c0 con 0x7f5e080a1a10 2024-09-15T03:58:46.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f10ccf39640 1 -- 172.21.15.110:0/3370070282 learned_addr learned my addr 172.21.15.110:0/3370070282 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:46.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e0dc68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e0dc68640 1 -- 172.21.15.110:0/3523483106 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e080a0850 con 0x7f5e0809ee80 2024-09-15T03:58:46.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e0dc68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e0dc68640 1 -- 172.21.15.110:0/3523483106 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0809c9a0 con 0x7f5e081369e0 2024-09-15T03:58:46.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e0dc68640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:46.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e0dc68640 10 monclient(hunting): authenticate will time out at 750.588548s 2024-09-15T03:58:46.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f10cf19d640 1 --2- 172.21.15.110:0/3370070282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10c809f800 0x7f10c806ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e07fff640 1 --2- 172.21.15.110:0/3523483106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e081369e0 0x7f5e08138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f10cf19d640 10 monclient(hunting): picked mon.noname-b con 0x7f10c809f800 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:46.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e07fff640 10 monclient(hunting): get_auth_request con 0x7f5e081369e0 auth_method 0 2024-09-15T03:58:46.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e07fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f10cf19d640 1 --2- 172.21.15.110:0/3370070282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10c80a1a10 0x7f10c809ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e06ffd640 1 --2- 172.21.15.110:0/3523483106 >> v2:172.21.15.177:3300/0 conn(0x7f5e0809ee80 0x7f5e08131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f10c7fff640 1 --2- 172.21.15.110:0/3370070282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10c809f800 0x7f10c806ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f4bbac20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:46.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e06ffd640 10 monclient(hunting): get_auth_request con 0x7f5e0809ee80 auth_method 0 2024-09-15T03:58:46.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f4bbac20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:46.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.943+0000 7f5e06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f4bbac20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:46.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f5e07fff640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-09-15T03:58:46.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f5e07fff640 1 -- 172.21.15.110:0/3523483106 >> v2:172.21.15.177:3300/0 conn(0x7f5e0809ee80 msgr2=0x7f5e08131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:46.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f5e07fff640 1 --2- 172.21.15.110:0/3523483106 >> v2:172.21.15.177:3300/0 conn(0x7f5e0809ee80 0x7f5e08131b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:46.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f5e07fff640 1 -- 172.21.15.110:0/3523483106 >> v1:172.21.15.110:6789/0 conn(0x7f5e080a1a10 legacy=0x7f5e0809e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:46.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f5e07fff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:46.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f5e07fff640 1 monclient: found mon.noname-b 2024-09-15T03:58:46.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f4bbac20640 1 -- 172.21.15.110:0/1145755548 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb40647e0 con 0x7f4bb40a1a10 2024-09-15T03:58:46.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f5e07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:46.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f5e07fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:46.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f5e07fff640 1 -- 172.21.15.110:0/3523483106 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0809cc80 con 0x7f5e081369e0 2024-09-15T03:58:46.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10cd73a640 1 --2- 172.21.15.110:0/3370070282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10c80a1a10 0x7f10c809ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10cf19d640 10 monclient(hunting): picked mon.noname-c con 0x7f10c80a1a10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:46.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10cf19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10cf19d640 1 -- 172.21.15.110:0/3370070282 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c806da40 con 0x7f10c809e770 2024-09-15T03:58:46.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10cf19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10cf19d640 1 -- 172.21.15.110:0/3370070282 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c806dcf0 con 0x7f10c809f800 2024-09-15T03:58:46.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10cf19d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:46.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10cf19d640 1 -- 172.21.15.110:0/3370070282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c806dfd0 con 0x7f10c80a1a10 2024-09-15T03:58:46.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10cf19d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:46.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10cf19d640 10 monclient(hunting): authenticate will time out at 750.589673s 2024-09-15T03:58:46.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10ccf39640 10 monclient(hunting): get_auth_request con 0x7f10c809e770 auth_method 0 2024-09-15T03:58:46.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f4bb0ff9640 1 -- 172.21.15.110:0/1145755548 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f4ba403c610 con 0x7f4bb40a1a10 2024-09-15T03:58:46.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10ccf39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10ccf39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10ccf39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10c7fff640 10 monclient(hunting): get_auth_request con 0x7f10c809f800 auth_method 0 2024-09-15T03:58:46.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10cd73a640 10 monclient(hunting): get_auth_request con 0x7f10c80a1a10 auth_method 0 2024-09-15T03:58:46.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10cd73a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:46.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10cd73a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:46.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10cd73a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:46.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.944+0000 7f10c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10cd73a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10cd73a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10cd73a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10ccf39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:46.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10ccf39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:46.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10ccf39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:46.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f4bb0ff9640 1 --2- 172.21.15.110:0/1145755548 >> v2:172.21.15.177:6812/28201 conn(0x7f4ba0038750 0x7f4ba003ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:46.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f4bb0ff9640 1 -- 172.21.15.110:0/1145755548 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f4ba406baf0 con 0x7f4bb40a1a10 2024-09-15T03:58:46.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10cd73a640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-09-15T03:58:46.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f4bb89bc640 1 --2- 172.21.15.110:0/1145755548 >> v2:172.21.15.177:6812/28201 conn(0x7f4ba0038750 0x7f4ba003ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:46.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f4bb89bc640 10 monclient: get_auth_request con 0x7f4ba0038750 auth_method 0 2024-09-15T03:58:47.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10cd73a640 1 -- 172.21.15.110:0/3370070282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10c809f800 msgr2=0x7f10c806ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10cd73a640 1 --2- 172.21.15.110:0/3370070282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10c809f800 0x7f10c806ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10cd73a640 1 -- 172.21.15.110:0/3370070282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10c809e770 msgr2=0x7f10c806c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10cd73a640 1 --2- 172.21.15.110:0/3370070282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10c809e770 0x7f10c806c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10cd73a640 10 monclient: _finish_hunting 0 2024-09-15T03:58:47.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10cd73a640 1 monclient: found mon.noname-c 2024-09-15T03:58:47.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10cd73a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:47.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10cd73a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:47.000 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10cd73a640 1 -- 172.21.15.110:0/3370070282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c0009a20 con 0x7f10c80a1a10 2024-09-15T03:58:47.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10c7fff640 1 --2- 172.21.15.110:0/3370070282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10c809f800 0x7f10c806ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:47.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.945+0000 7f10ccf39640 1 --2- 172.21.15.110:0/3370070282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10c809e770 0x7f10c806c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:47.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.946+0000 7f5e07fff640 1 -- 172.21.15.110:0/3523483106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e081369e0 msgr2=0x7f5e08138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:47.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.946+0000 7f5e07fff640 1 --2- 172.21.15.110:0/3523483106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e081369e0 0x7f5e08138dc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5e00009e60 tx=0x7f5e00009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb1bf6e88571b989 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:47.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e067fc640 1 -- 172.21.15.110:0/3523483106 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e00007760 con 0x7f5e081369e0 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e067fc640 10 monclient: dump: 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e067fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:47.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e0dc68640 5 monclient: authenticate success, global_id 4292 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e0dc68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e067fc640 1 -- 172.21.15.110:0/3523483106 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0002d050 con 0x7f5e081369e0 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e0dc68640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e0dc68640 4 set_mon_vals no callback set 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e067fc640 1 -- 172.21.15.110:0/3523483106 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e00007b30 con 0x7f5e081369e0 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f5e067fc640 10 monclient: dump: 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:47.003 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:47.004 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:47.004 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:47.004 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:47.004 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:47.004 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:47.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10cd73a640 1 -- 172.21.15.110:0/3370070282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10c80a1a10 msgr2=0x7f10c809ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:47.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10cd73a640 1 --2- 172.21.15.110:0/3370070282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10c80a1a10 0x7f10c809ae30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009f10 tx=0x7f10c000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:47.004 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10c4ff9640 1 -- 172.21.15.110:0/3370070282 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0014480 con 0x7f10c80a1a10 2024-09-15T03:58:47.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:47.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f4bb89bc640 1 --2- 172.21.15.110:0/1145755548 >> v2:172.21.15.177:6812/28201 conn(0x7f4ba0038750 0x7f4ba003ac00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80037d0 tx=0x7f4ba8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:47.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.948+0000 7f5e0dc68640 10 monclient: shutdown 2024-09-15T03:58:47.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.948+0000 7f5e0dc68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:47.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.948+0000 7f5e0dc68640 1 -- 172.21.15.110:0/3523483106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e081369e0 msgr2=0x7f5e08138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.948+0000 7f5e0dc68640 1 --2- 172.21.15.110:0/3523483106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e081369e0 0x7f5e08138dc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5e00009e60 tx=0x7f5e00009780 comp rx=0 tx=0).stop 2024-09-15T03:58:47.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10c4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:47.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10c4ff9640 10 monclient: dump: 2024-09-15T03:58:47.006 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:47.006 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:47.006 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:47.006 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:47.006 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:47.006 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:47.006 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:47.006 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:47.006 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:47.006 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:47.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10c4ff9640 10 monclient: _finish_auth 0 2024-09-15T03:58:47.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:47.006 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10c4ff9640 1 -- 172.21.15.110:0/3370070282 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c000f040 con 0x7f10c80a1a10 2024-09-15T03:58:47.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:47.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10c4ff9640 1 -- 172.21.15.110:0/3370070282 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0014830 con 0x7f10c80a1a10 2024-09-15T03:58:47.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:47.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10c4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:47.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10c4ff9640 10 monclient: dump: 2024-09-15T03:58:47.007 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:47.007 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:47.007 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:47.007 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:47.007 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:47.007 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:47.007 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:47.007 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:47.007 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:47.008 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:47.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10cf19d640 5 monclient: authenticate success, global_id 4295 2024-09-15T03:58:47.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10c6ffd640 4 set_mon_vals no callback set 2024-09-15T03:58:47.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10cf19d640 10 monclient: _renew_subs 2024-09-15T03:58:47.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10cf19d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:47.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10cf19d640 1 -- 172.21.15.110:0/3370070282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10bc009390 con 0x7f10c80a1a10 2024-09-15T03:58:47.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10cf19d640 10 monclient: _renew_subs 2024-09-15T03:58:47.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10cf19d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:47.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.947+0000 7f10cf19d640 1 -- 172.21.15.110:0/3370070282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b4009450 con 0x7f10c80a1a10 2024-09-15T03:58:47.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.948+0000 7f5e0dc68640 1 -- 172.21.15.110:0/3523483106 shutdown_connections 2024-09-15T03:58:47.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.948+0000 7f5e0dc68640 1 --2- 172.21.15.110:0/3523483106 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e081369e0 0x7f5e08138dc0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.948+0000 7f5e0dc68640 1 --2- 172.21.15.110:0/3523483106 >> v2:172.21.15.177:3300/0 conn(0x7f5e0809ee80 0x7f5e08131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.948+0000 7f5e0dc68640 1 -- 172.21.15.110:0/3523483106 >> 172.21.15.110:0/3523483106 conn(0x7f5e0806e860 msgr2=0x7f5e0806ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:47.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.948+0000 7f5e0dc68640 1 -- 172.21.15.110:0/3523483106 shutdown_connections 2024-09-15T03:58:47.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.949+0000 7f10cf19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:47.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.949+0000 7f10cf19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:47.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.949+0000 7f10cf19d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:47.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.949+0000 7f10cf19d640 1 -- 172.21.15.110:0/3370070282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c806b440 con 0x7f10c80a1a10 2024-09-15T03:58:47.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.948+0000 7f5e0dc68640 1 -- 172.21.15.110:0/3523483106 wait complete. 2024-09-15T03:58:47.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.949+0000 7f5e0dc68640 10 monclient: build_initial_monmap 2024-09-15T03:58:47.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.949+0000 7f5e0dc68640 10 monclient: monmap: 2024-09-15T03:58:47.010 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:47.010 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:47.010 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:47.010 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:47.010 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:47.010 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:47.011 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:47.011 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:47.011 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:47.011 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:47.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.949+0000 7f5e0dc68640 1 Processor -- start 2024-09-15T03:58:47.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.949+0000 7f5e0dc68640 1 -- start start 2024-09-15T03:58:47.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.949+0000 7f5e0dc68640 10 monclient: init 2024-09-15T03:58:47.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.949+0000 7f10c4ff9640 1 -- 172.21.15.110:0/3370070282 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f10c0003020 con 0x7f10c80a1a10 2024-09-15T03:58:47.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.949+0000 7f10c4ff9640 1 --2- 172.21.15.110:0/3370070282 >> v2:172.21.15.177:6812/28201 conn(0x7f10ac038750 0x7f10ac03ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:47.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f10c4ff9640 1 -- 172.21.15.110:0/3370070282 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f10c004c190 con 0x7f10c80a1a10 2024-09-15T03:58:47.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f10ccf39640 1 --2- 172.21.15.110:0/3370070282 >> v2:172.21.15.177:6812/28201 conn(0x7f10ac038750 0x7f10ac03ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:47.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f10ccf39640 10 monclient: get_auth_request con 0x7f10ac038750 auth_method 0 2024-09-15T03:58:47.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:47.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:58:47.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e0809ee80 0x7f5e0814d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:47.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 10 monclient(hunting): picked mon.noname-b con 0x7f5e0809ee80 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:47.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e080a1a10 0x7f5e08130820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:47.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e06ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e080a1a10 0x7f5e08130820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:47.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e06ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e080a1a10 0x7f5e08130820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55138/0 (socket says 172.21.15.110:55138) 2024-09-15T03:58:47.013 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e077fe640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e0809ee80 0x7f5e0814d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:47.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e06ffd640 1 -- 172.21.15.110:0/625069025 learned_addr learned my addr 172.21.15.110:0/625069025 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:47.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 10 monclient(hunting): picked mon.noname-c con 0x7f5e080a1a10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:47.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 1 --2- 172.21.15.110:0/625069025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e081369e0 0x7f5e0806b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:47.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 10 monclient(hunting): picked mon.noname-a con 0x7f5e081369e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:47.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:47.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 1 -- 172.21.15.110:0/625069025 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0806cf80 con 0x7f5e081369e0 2024-09-15T03:58:47.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:47.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 1 -- 172.21.15.110:0/625069025 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0812ca30 con 0x7f5e0809ee80 2024-09-15T03:58:47.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:47.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 1 -- 172.21.15.110:0/625069025 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0812cff0 con 0x7f5e080a1a10 2024-09-15T03:58:47.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:47.014 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e07fff640 1 --2- 172.21.15.110:0/625069025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e081369e0 0x7f5e0806b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:47.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f4bb0ff9640 1 -- 172.21.15.110:0/1145755548 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4ba4038890 con 0x7f4bb40a1a10 2024-09-15T03:58:47.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f4bb0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:47.015 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f4bb0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:47.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e0dc68640 10 monclient(hunting): authenticate will time out at 750.595851s 2024-09-15T03:58:47.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e06ffd640 10 monclient(hunting): get_auth_request con 0x7f5e080a1a10 auth_method 0 2024-09-15T03:58:47.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:47.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:47.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:47.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e077fe640 10 monclient(hunting): get_auth_request con 0x7f5e0809ee80 auth_method 0 2024-09-15T03:58:47.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e077fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e07fff640 10 monclient(hunting): get_auth_request con 0x7f5e081369e0 auth_method 0 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e07fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.950+0000 7f5e07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e077fe640 10 monclient(hunting): handle_auth_done global_id 4288 payload 824 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e077fe640 1 -- 172.21.15.110:0/625069025 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e080a1a10 msgr2=0x7f5e08130820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e077fe640 1 --2- 172.21.15.110:0/625069025 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e080a1a10 0x7f5e08130820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e077fe640 1 -- 172.21.15.110:0/625069025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e081369e0 msgr2=0x7f5e0806b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e077fe640 1 --2- 172.21.15.110:0/625069025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e081369e0 0x7f5e0806b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e077fe640 10 monclient: _finish_hunting 0 2024-09-15T03:58:47.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e077fe640 1 monclient: found mon.noname-b 2024-09-15T03:58:47.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:47.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e077fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.177:3300/0 2024-09-15T03:58:47.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e077fe640 1 -- 172.21.15.110:0/625069025 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e00009a20 con 0x7f5e0809ee80 2024-09-15T03:58:47.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e07fff640 1 --2- 172.21.15.110:0/625069025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e081369e0 0x7f5e0806b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:47.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.951+0000 7f5e06ffd640 1 --2- 172.21.15.110:0/625069025 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e080a1a10 0x7f5e08130820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:58:47.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.953+0000 7f10ccf39640 1 --2- 172.21.15.110:0/3370070282 >> v2:172.21.15.177:6812/28201 conn(0x7f10ac038750 0x7f10ac03ac00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f10bc011640 tx=0x7f10bc0090f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:47.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.953+0000 7f5e077fe640 1 -- 172.21.15.110:0/625069025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e0809ee80 msgr2=0x7f5e0814d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:47.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.953+0000 7f5e077fe640 1 --2- 172.21.15.110:0/625069025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e0809ee80 0x7f5e0814d740 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5df800c490 tx=0x7f5df8007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:47.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.953+0000 7f5def7fe640 1 -- 172.21.15.110:0/625069025 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df8005cb0 con 0x7f5e0809ee80 2024-09-15T03:58:47.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.953+0000 7f5def7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:47.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5def7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:47.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f10c4ff9640 1 -- 172.21.15.110:0/3370070282 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f10c00114f0 con 0x7f10c80a1a10 2024-09-15T03:58:47.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f10c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:47.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f10c4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:47.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5def7fe640 10 monclient: dump: 2024-09-15T03:58:47.021 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:47.021 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:47.021 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:47.021 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:47.021 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:47.021 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:47.022 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:47.022 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:47.022 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:47.022 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:47.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5def7fe640 10 monclient: _finish_auth 0 2024-09-15T03:58:47.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5def7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:47.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5e0dc68640 5 monclient: authenticate success, global_id 4288 2024-09-15T03:58:47.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5def7fe640 1 -- 172.21.15.110:0/625069025 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df8005e30 con 0x7f5e0809ee80 2024-09-15T03:58:47.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5def7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:47.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5def7fe640 1 -- 172.21.15.110:0/625069025 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df8007300 con 0x7f5e0809ee80 2024-09-15T03:58:47.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5e0dc68640 10 monclient: _renew_subs 2024-09-15T03:58:47.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5e0dc68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:47.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5e0dc68640 1 -- 172.21.15.110:0/625069025 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0812cd10 con 0x7f5e0809ee80 2024-09-15T03:58:47.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5def7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:47.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5def7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:47.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5def7fe640 10 monclient: dump: 2024-09-15T03:58:47.023 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:47.023 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:47.023 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:47.023 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:47.023 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:47.023 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:47.023 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:47.023 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:47.023 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:47.023 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:47.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5e05ffb640 4 set_mon_vals no callback set 2024-09-15T03:58:47.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5e0dc68640 10 monclient: _renew_subs 2024-09-15T03:58:47.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5e0dc68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:47.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.954+0000 7f5e0dc68640 1 -- 172.21.15.110:0/625069025 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0814dc80 con 0x7f5e0809ee80 2024-09-15T03:58:47.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.955+0000 7f5def7fe640 1 -- 172.21.15.110:0/625069025 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f5df801b3b0 con 0x7f5e0809ee80 2024-09-15T03:58:47.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.955+0000 7f5e0dc68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:47.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.955+0000 7f5e0dc68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:47.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.955+0000 7f5e0dc68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:47.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.955+0000 7f5e0dc68640 1 -- 172.21.15.110:0/625069025 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e080647e0 con 0x7f5e0809ee80 2024-09-15T03:58:47.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.955+0000 7f5def7fe640 1 --2- 172.21.15.110:0/625069025 >> v2:172.21.15.177:6812/28201 conn(0x7f5df4038700 0x7f5df403abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:47.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.955+0000 7f5def7fe640 1 -- 172.21.15.110:0/625069025 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3558+0+0 (secure 0 0 0) 0x7f5df804aab0 con 0x7f5e0809ee80 2024-09-15T03:58:47.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.956+0000 7f5e06ffd640 1 --2- 172.21.15.110:0/625069025 >> v2:172.21.15.177:6812/28201 conn(0x7f5df4038700 0x7f5df403abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:47.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.956+0000 7f5e06ffd640 10 monclient: get_auth_request con 0x7f5df4038700 auth_method 0 2024-09-15T03:58:47.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.958+0000 7f5e06ffd640 1 --2- 172.21.15.110:0/625069025 >> v2:172.21.15.177:6812/28201 conn(0x7f5df4038700 0x7f5df403abb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5e080a2440 tx=0x7f5dfc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:47.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.960+0000 7f5def7fe640 1 -- 172.21.15.110:0/625069025 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5df801a360 con 0x7f5e0809ee80 2024-09-15T03:58:47.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.960+0000 7f5def7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:47.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:46.960+0000 7f5def7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:47.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.019+0000 7f59d4c89640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:58:47.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.019+0000 7f59d4c89640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:58:47.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.019+0000 7f59d4c89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:47.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.019+0000 7f59d4c89640 1 -- 172.21.15.110:0/695862435 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f59d0067130 con 0x7f59d0136050 2024-09-15T03:58:47.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.019+0000 7f59be7fc640 1 -- 172.21.15.110:0/695862435 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f59c401a360 con 0x7f59d0136050 2024-09-15T03:58:47.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.019+0000 7f59be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:58:47.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.019+0000 7f59be7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:47.026 INFO:teuthology.orchestra.run.smithi110.stdout:47244640258 2024-09-15T03:58:47.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.022+0000 7f59d4c89640 1 -- 172.21.15.110:0/695862435 >> v2:172.21.15.177:6812/28201 conn(0x7f59b0038750 msgr2=0x7f59b003ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.022+0000 7f59d4c89640 1 --2- 172.21.15.110:0/695862435 >> v2:172.21.15.177:6812/28201 conn(0x7f59b0038750 0x7f59b003ac00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f59c0007fd0 tx=0x7f59c0007a50 comp rx=0 tx=0).stop 2024-09-15T03:58:47.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.022+0000 7f59d4c89640 10 monclient: shutdown 2024-09-15T03:58:47.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.022+0000 7f59d4c89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:47.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.022+0000 7f59d4c89640 1 -- 172.21.15.110:0/695862435 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59d0136050 msgr2=0x7f59d0130760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.022+0000 7f59d4c89640 1 --2- 172.21.15.110:0/695862435 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59d0136050 0x7f59d0130760 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f59c400c7c0 tx=0x7f59c4001b30 comp rx=0 tx=0).stop 2024-09-15T03:58:47.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.022+0000 7f59d4c89640 1 -- 172.21.15.110:0/695862435 shutdown_connections 2024-09-15T03:58:47.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.022+0000 7f59d4c89640 1 --2- 172.21.15.110:0/695862435 >> v2:172.21.15.177:6812/28201 conn(0x7f59b0038750 0x7f59b003ac00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.022+0000 7f59d4c89640 1 --2- 172.21.15.110:0/695862435 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f59d0130ca0 0x7f59d011b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.022+0000 7f59d4c89640 1 --2- 172.21.15.110:0/695862435 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59d0136050 0x7f59d0130760 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.022+0000 7f59d4c89640 1 --2- 172.21.15.110:0/695862435 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59b80051f0 0x7f59d009a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.022+0000 7f59d4c89640 1 -- 172.21.15.110:0/695862435 >> 172.21.15.110:0/695862435 conn(0x7f59d006e860 msgr2=0x7f59d00a2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:47.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.022+0000 7f59d4c89640 1 -- 172.21.15.110:0/695862435 shutdown_connections 2024-09-15T03:58:47.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.022+0000 7f59d4c89640 1 -- 172.21.15.110:0/695862435 wait complete. 2024-09-15T03:58:47.034 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2024-09-15T03:58:47.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.088+0000 7f25d059d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:58:47.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.088+0000 7f25d059d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:58:47.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.088+0000 7f25d059d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:47.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.088+0000 7f25d059d640 1 -- 172.21.15.110:0/1150490368 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f25c80647e0 con 0x7f25c81369e0 2024-09-15T03:58:47.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.089+0000 7f25be7fc640 1 -- 172.21.15.110:0/1150490368 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f25c0039300 con 0x7f25c81369e0 2024-09-15T03:58:47.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.089+0000 7f25be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:58:47.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.089+0000 7f25be7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:47.091 INFO:teuthology.orchestra.run.smithi110.stdout:47244640260 2024-09-15T03:58:47.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.090+0000 7f25d059d640 1 -- 172.21.15.110:0/1150490368 >> v2:172.21.15.177:6812/28201 conn(0x7f25ac038750 msgr2=0x7f25ac03ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.090+0000 7f25d059d640 1 --2- 172.21.15.110:0/1150490368 >> v2:172.21.15.177:6812/28201 conn(0x7f25ac038750 0x7f25ac03ac00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f25c4007fd0 tx=0x7f25c4007e10 comp rx=0 tx=0).stop 2024-09-15T03:58:47.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.091+0000 7f25d059d640 10 monclient: shutdown 2024-09-15T03:58:47.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.091+0000 7f25d059d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:47.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.091+0000 7f25d059d640 1 -- 172.21.15.110:0/1150490368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25c81369e0 msgr2=0x7f25c8095fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.091+0000 7f25d059d640 1 --2- 172.21.15.110:0/1150490368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25c81369e0 0x7f25c8095fb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f25c0014b20 tx=0x7f25c003d000 comp rx=0 tx=0).stop 2024-09-15T03:58:47.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.091+0000 7f25d059d640 1 -- 172.21.15.110:0/1150490368 shutdown_connections 2024-09-15T03:58:47.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.091+0000 7f25d059d640 1 --2- 172.21.15.110:0/1150490368 >> v2:172.21.15.177:6812/28201 conn(0x7f25ac038750 0x7f25ac03ac00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.091+0000 7f25d059d640 1 --2- 172.21.15.110:0/1150490368 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f25c81369e0 0x7f25c8095fb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.091+0000 7f25d059d640 1 --2- 172.21.15.110:0/1150490368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f25c80a1a10 0x7f25c8095a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.091+0000 7f25d059d640 1 --2- 172.21.15.110:0/1150490368 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f25c809ee80 0x7f25c8135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.091+0000 7f25d059d640 1 -- 172.21.15.110:0/1150490368 >> 172.21.15.110:0/1150490368 conn(0x7f25c806e860 msgr2=0x7f25c809e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:47.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.091+0000 7f25d059d640 1 -- 172.21.15.110:0/1150490368 shutdown_connections 2024-09-15T03:58:47.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.091+0000 7f25d059d640 1 -- 172.21.15.110:0/1150490368 wait complete. 2024-09-15T03:58:47.103 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.0 2024-09-15T03:58:47.103 DEBUG:teuthology.parallel:result is None 2024-09-15T03:58:47.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.152+0000 7f4bbac20640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:58:47.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.152+0000 7f4bbac20640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:58:47.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.152+0000 7f4bbac20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:47.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.152+0000 7f4bbac20640 1 -- 172.21.15.110:0/1145755548 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4bb4067130 con 0x7f4bb40a1a10 2024-09-15T03:58:47.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.152+0000 7f4bb0ff9640 1 -- 172.21.15.110:0/1145755548 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4ba4037090 con 0x7f4bb40a1a10 2024-09-15T03:58:47.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.152+0000 7f4bb0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:58:47.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.152+0000 7f4bb0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:47.155 INFO:teuthology.orchestra.run.smithi110.stdout:47244640260 2024-09-15T03:58:47.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.154+0000 7f4bbac20640 1 -- 172.21.15.110:0/1145755548 >> v2:172.21.15.177:6812/28201 conn(0x7f4ba0038750 msgr2=0x7f4ba003ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.154+0000 7f4bbac20640 1 --2- 172.21.15.110:0/1145755548 >> v2:172.21.15.177:6812/28201 conn(0x7f4ba0038750 0x7f4ba003ac00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ba80037d0 tx=0x7f4ba8009040 comp rx=0 tx=0).stop 2024-09-15T03:58:47.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.154+0000 7f4bbac20640 10 monclient: shutdown 2024-09-15T03:58:47.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.154+0000 7f4bbac20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:47.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.154+0000 7f4bbac20640 1 -- 172.21.15.110:0/1145755548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bb40a1a10 msgr2=0x7f4bb40971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.154+0000 7f4bbac20640 1 --2- 172.21.15.110:0/1145755548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bb40a1a10 0x7f4bb40971e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009940 tx=0x7f4ba4032000 comp rx=0 tx=0).stop 2024-09-15T03:58:47.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.154+0000 7f4bbac20640 1 -- 172.21.15.110:0/1145755548 shutdown_connections 2024-09-15T03:58:47.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.154+0000 7f4bbac20640 1 --2- 172.21.15.110:0/1145755548 >> v2:172.21.15.177:6812/28201 conn(0x7f4ba0038750 0x7f4ba003ac00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.154+0000 7f4bbac20640 1 --2- 172.21.15.110:0/1145755548 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4bb41369e0 0x7f4bb4097720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.154+0000 7f4bbac20640 1 --2- 172.21.15.110:0/1145755548 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4bb40a1a10 0x7f4bb40971e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.154+0000 7f4bbac20640 1 --2- 172.21.15.110:0/1145755548 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4bb409ee80 0x7f4bb409b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.154+0000 7f4bbac20640 1 -- 172.21.15.110:0/1145755548 >> 172.21.15.110:0/1145755548 conn(0x7f4bb406e860 msgr2=0x7f4bb40a3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:47.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.154+0000 7f4bbac20640 1 -- 172.21.15.110:0/1145755548 shutdown_connections 2024-09-15T03:58:47.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.154+0000 7f4bbac20640 1 -- 172.21.15.110:0/1145755548 wait complete. 2024-09-15T03:58:47.166 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.4 2024-09-15T03:58:47.166 DEBUG:teuthology.parallel:result is None 2024-09-15T03:58:47.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.170+0000 7f10cf19d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:58:47.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.170+0000 7f10cf19d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:58:47.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.170+0000 7f10cf19d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:47.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.170+0000 7f10cf19d640 1 -- 172.21.15.110:0/3370070282 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f10c80647e0 con 0x7f10c80a1a10 2024-09-15T03:58:47.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.170+0000 7f10c4ff9640 1 -- 172.21.15.110:0/3370070282 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f10c0018020 con 0x7f10c80a1a10 2024-09-15T03:58:47.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.171+0000 7f10c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:58:47.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.171+0000 7f10c4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:47.173 INFO:teuthology.orchestra.run.smithi110.stdout:47244640260 2024-09-15T03:58:47.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.174+0000 7f10cf19d640 1 -- 172.21.15.110:0/3370070282 >> v2:172.21.15.177:6812/28201 conn(0x7f10ac038750 msgr2=0x7f10ac03ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.174+0000 7f10cf19d640 1 --2- 172.21.15.110:0/3370070282 >> v2:172.21.15.177:6812/28201 conn(0x7f10ac038750 0x7f10ac03ac00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f10bc011640 tx=0x7f10bc0090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.174+0000 7f10cf19d640 10 monclient: shutdown 2024-09-15T03:58:47.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.174+0000 7f10cf19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:47.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.174+0000 7f10cf19d640 1 -- 172.21.15.110:0/3370070282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10c80a1a10 msgr2=0x7f10c809ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.174+0000 7f10cf19d640 1 --2- 172.21.15.110:0/3370070282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10c80a1a10 0x7f10c809ae30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009f10 tx=0x7f10c000b040 comp rx=0 tx=0).stop 2024-09-15T03:58:47.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.174+0000 7f10cf19d640 1 -- 172.21.15.110:0/3370070282 shutdown_connections 2024-09-15T03:58:47.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.174+0000 7f10cf19d640 1 --2- 172.21.15.110:0/3370070282 >> v2:172.21.15.177:6812/28201 conn(0x7f10ac038750 0x7f10ac03ac00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.174+0000 7f10cf19d640 1 --2- 172.21.15.110:0/3370070282 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f10c80a1a10 0x7f10c809ae30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.174+0000 7f10cf19d640 1 --2- 172.21.15.110:0/3370070282 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f10c809f800 0x7f10c806ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.174+0000 7f10cf19d640 1 --2- 172.21.15.110:0/3370070282 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10c809e770 0x7f10c806c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.174+0000 7f10cf19d640 1 -- 172.21.15.110:0/3370070282 >> 172.21.15.110:0/3370070282 conn(0x7f10c806e860 msgr2=0x7f10c80a4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:47.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.174+0000 7f10cf19d640 1 -- 172.21.15.110:0/3370070282 shutdown_connections 2024-09-15T03:58:47.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.174+0000 7f10cf19d640 1 -- 172.21.15.110:0/3370070282 wait complete. 2024-09-15T03:58:47.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.176+0000 7f5e0dc68640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:58:47.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.176+0000 7f5e0dc68640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:58:47.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.176+0000 7f5e0dc68640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.177:3300/0 2024-09-15T03:58:47.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.176+0000 7f5e0dc68640 1 -- 172.21.15.110:0/625069025 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5e08067130 con 0x7f5e0809ee80 2024-09-15T03:58:47.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.176+0000 7f5def7fe640 1 -- 172.21.15.110:0/625069025 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5df8017030 con 0x7f5e0809ee80 2024-09-15T03:58:47.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.176+0000 7f5def7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:58:47.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.176+0000 7f5def7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:47.179 INFO:teuthology.orchestra.run.smithi110.stdout:47244640260 2024-09-15T03:58:47.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.180+0000 7f5e0dc68640 1 -- 172.21.15.110:0/625069025 >> v2:172.21.15.177:6812/28201 conn(0x7f5df4038700 msgr2=0x7f5df403abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.180+0000 7f5e0dc68640 1 --2- 172.21.15.110:0/625069025 >> v2:172.21.15.177:6812/28201 conn(0x7f5df4038700 0x7f5df403abb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5e080a2440 tx=0x7f5dfc008040 comp rx=0 tx=0).stop 2024-09-15T03:58:47.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.180+0000 7f5e0dc68640 10 monclient: shutdown 2024-09-15T03:58:47.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.180+0000 7f5e0dc68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:47.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.180+0000 7f5e0dc68640 1 -- 172.21.15.110:0/625069025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e0809ee80 msgr2=0x7f5e0814d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:47.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.180+0000 7f5e0dc68640 1 --2- 172.21.15.110:0/625069025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e0809ee80 0x7f5e0814d740 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5df800c490 tx=0x7f5df8007f00 comp rx=0 tx=0).stop 2024-09-15T03:58:47.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.180+0000 7f5e0dc68640 1 -- 172.21.15.110:0/625069025 shutdown_connections 2024-09-15T03:58:47.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.180+0000 7f5e0dc68640 1 --2- 172.21.15.110:0/625069025 >> v2:172.21.15.177:6812/28201 conn(0x7f5df4038700 0x7f5df403abb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.180+0000 7f5e0dc68640 1 --2- 172.21.15.110:0/625069025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e081369e0 0x7f5e0806b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.180+0000 7f5e0dc68640 1 --2- 172.21.15.110:0/625069025 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5e080a1a10 0x7f5e08130820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.180+0000 7f5e0dc68640 1 --2- 172.21.15.110:0/625069025 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5e0809ee80 0x7f5e0814d740 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:47.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.180+0000 7f5e0dc68640 1 -- 172.21.15.110:0/625069025 >> 172.21.15.110:0/625069025 conn(0x7f5e0806e860 msgr2=0x7f5e080a4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:47.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.180+0000 7f5e0dc68640 1 -- 172.21.15.110:0/625069025 shutdown_connections 2024-09-15T03:58:47.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:47.180+0000 7f5e0dc68640 1 -- 172.21.15.110:0/625069025 wait complete. 2024-09-15T03:58:47.189 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.2 2024-09-15T03:58:47.189 DEBUG:teuthology.parallel:result is None 2024-09-15T03:58:47.193 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.1 2024-09-15T03:58:47.193 DEBUG:teuthology.parallel:result is None 2024-09-15T03:58:47.985 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T03:58:48.035 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:58:48.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.087+0000 7f0e0f31d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:48.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.087+0000 7f0e0f31d640 10 monclient: build_initial_monmap 2024-09-15T03:58:48.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.087+0000 7f0e0f31d640 10 monclient: monmap: 2024-09-15T03:58:48.088 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:48.088 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.088 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:48.088032+0000 2024-09-15T03:58:48.088 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:48.088032+0000 2024-09-15T03:58:48.088 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:48.088 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.088 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:48.088 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:48.089 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:48.089 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:48.089 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:48.089 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.087+0000 7f0e0f31d640 1 Processor -- start 2024-09-15T03:58:48.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0f31d640 1 -- start start 2024-09-15T03:58:48.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0f31d640 10 monclient: init 2024-09-15T03:58:48.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0f31d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:48.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0f31d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T03:58:48.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0f31d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 0x7f0e0809f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0f31d640 10 monclient(hunting): picked mon.noname-b con 0x7f0e0809f180 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:48.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0f31d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e0809fa90 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:48.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0f31d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e081331f0 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:48.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0f31d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e080a05b0 con 0x7f0e081331f0 2024-09-15T03:58:48.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0f31d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0806aaf0 con 0x7f0e0809fa90 2024-09-15T03:58:48.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0f31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0f31d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0806d160 con 0x7f0e0809f180 2024-09-15T03:58:48.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0f31d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:48.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0f31d640 10 monclient(hunting): authenticate will time out at 751.732853s 2024-09-15T03:58:48.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0d0b9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 0x7f0e0809f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0d0b9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 0x7f0e0809f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55150/0 (socket says 172.21.15.110:55150) 2024-09-15T03:58:48.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0d0b9640 1 -- 172.21.15.110:0/666034945 learned_addr learned my addr 172.21.15.110:0/666034945 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:48.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0d0b9640 10 monclient(hunting): get_auth_request con 0x7f0e0809f180 auth_method 0 2024-09-15T03:58:48.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0d0b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0d0b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.088+0000 7f0e0d0b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.089+0000 7f0e0d0b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.089+0000 7f0e0d0b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.089+0000 7f0e0d0b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.089+0000 7f0e0d0b9640 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2024-09-15T03:58:48.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.089+0000 7f0e0d0b9640 1 -- 172.21.15.110:0/666034945 >> v1:172.21.15.177:6789/0 conn(0x7f0e0809fa90 legacy=0x7f0e0809fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:48.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.089+0000 7f0e0d0b9640 1 -- 172.21.15.110:0/666034945 >> v1:172.21.15.110:6789/0 conn(0x7f0e081331f0 legacy=0x7f0e081355d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:48.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.089+0000 7f0e0d0b9640 10 monclient: _finish_hunting 0 2024-09-15T03:58:48.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.089+0000 7f0e0d0b9640 1 monclient: found mon.noname-b 2024-09-15T03:58:48.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.089+0000 7f0e0d0b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:48.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.089+0000 7f0e0d0b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.089+0000 7f0e0d0b9640 1 -- 172.21.15.110:0/666034945 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0806acf0 con 0x7f0e0809f180 2024-09-15T03:58:48.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0e0d0b9640 1 -- 172.21.15.110:0/666034945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 msgr2=0x7f0e0809f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:48.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0e0d0b9640 1 --2- 172.21.15.110:0/666034945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 0x7f0e0809f550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009970 tx=0x7f0e040091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=901ea81ad8e4c82e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:48.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0dfffff640 1 -- 172.21.15.110:0/666034945 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e040304b0 con 0x7f0e0809f180 2024-09-15T03:58:48.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0dfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0dfffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:48.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0dfffff640 10 monclient: dump: 2024-09-15T03:58:48.100 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.100 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.100 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.100 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.100 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.100 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.100 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.100 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.100 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.100 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0dfffff640 10 monclient: _finish_auth 0 2024-09-15T03:58:48.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0dfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:48.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0dfffff640 1 -- 172.21.15.110:0/666034945 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e04030610 con 0x7f0e0809f180 2024-09-15T03:58:48.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0e0f31d640 5 monclient: authenticate success, global_id 4301 2024-09-15T03:58:48.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0e0f31d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:48.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0dfffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:48.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0e0f31d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:48.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0dfffff640 1 -- 172.21.15.110:0/666034945 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e040309b0 con 0x7f0e0809f180 2024-09-15T03:58:48.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0e0f31d640 4 set_mon_vals no callback set 2024-09-15T03:58:48.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0dfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0dfffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:48.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0dfffff640 10 monclient: dump: 2024-09-15T03:58:48.101 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.101 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.102 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.102 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.102 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.102 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.102 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.102 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.102 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.102 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0e0f31d640 10 monclient: shutdown 2024-09-15T03:58:48.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0e0f31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:48.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0e0f31d640 1 -- 172.21.15.110:0/666034945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 msgr2=0x7f0e0809f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0e0f31d640 1 --2- 172.21.15.110:0/666034945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 0x7f0e0809f550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009970 tx=0x7f0e040091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0e0f31d640 1 -- 172.21.15.110:0/666034945 shutdown_connections 2024-09-15T03:58:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0e0f31d640 1 --2- 172.21.15.110:0/666034945 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 0x7f0e0809f550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.092+0000 7f0e0f31d640 1 -- 172.21.15.110:0/666034945 >> 172.21.15.110:0/666034945 conn(0x7f0e0809d680 msgr2=0x7f0e0809da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.093+0000 7f0e0f31d640 1 -- 172.21.15.110:0/666034945 shutdown_connections 2024-09-15T03:58:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.093+0000 7f0e0f31d640 1 -- 172.21.15.110:0/666034945 wait complete. 2024-09-15T03:58:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.093+0000 7f0e0f31d640 10 monclient: build_initial_monmap 2024-09-15T03:58:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.093+0000 7f0e0f31d640 10 monclient: monmap: 2024-09-15T03:58:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.103 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:48.104 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.093+0000 7f0e0f31d640 1 Processor -- start 2024-09-15T03:58:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.093+0000 7f0e0f31d640 1 -- start start 2024-09-15T03:58:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.093+0000 7f0e0f31d640 10 monclient: init 2024-09-15T03:58:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:58:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 0x7f0e0812fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 10 monclient(hunting): picked mon.noname-c con 0x7f0e0809f180 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e0809fa90 0x7f0e08119430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 10 monclient(hunting): picked mon.noname-a con 0x7f0e0809fa90 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e081331f0 0x7f0e08119970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 10 monclient(hunting): picked mon.noname-b con 0x7f0e081331f0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0806b240 con 0x7f0e0809fa90 2024-09-15T03:58:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0811ca80 con 0x7f0e081331f0 2024-09-15T03:58:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0811b4d0 con 0x7f0e0809f180 2024-09-15T03:58:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0f31d640 10 monclient(hunting): authenticate will time out at 751.738575s 2024-09-15T03:58:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d0b9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 0x7f0e0812fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d0b9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 0x7f0e0812fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55162/0 (socket says 172.21.15.110:55162) 2024-09-15T03:58:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d0b9640 1 -- 172.21.15.110:0/1733398258 learned_addr learned my addr 172.21.15.110:0/1733398258 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d0b9640 10 monclient(hunting): get_auth_request con 0x7f0e0809f180 auth_method 0 2024-09-15T03:58:48.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d0b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d0b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d0b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0c8b8640 1 --2- 172.21.15.110:0/1733398258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e0809fa90 0x7f0e08119430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0c8b8640 10 monclient(hunting): get_auth_request con 0x7f0e0809fa90 auth_method 0 2024-09-15T03:58:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0c8b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0c8b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0c8b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d0b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d0b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d0b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d8ba640 1 --2- 172.21.15.110:0/1733398258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e081331f0 0x7f0e08119970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d8ba640 10 monclient(hunting): get_auth_request con 0x7f0e081331f0 auth_method 0 2024-09-15T03:58:48.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d8ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d8ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d8ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0c8b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0c8b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0c8b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d8ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d8ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d8ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.094+0000 7f0e0d0b9640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-09-15T03:58:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.095+0000 7f0e0d0b9640 1 -- 172.21.15.110:0/1733398258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e081331f0 msgr2=0x7f0e08119970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.095+0000 7f0e0d0b9640 1 --2- 172.21.15.110:0/1733398258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e081331f0 0x7f0e08119970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.095+0000 7f0e0d0b9640 1 -- 172.21.15.110:0/1733398258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e0809fa90 msgr2=0x7f0e08119430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.095+0000 7f0e0d0b9640 1 --2- 172.21.15.110:0/1733398258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e0809fa90 0x7f0e08119430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.095+0000 7f0e0d0b9640 10 monclient: _finish_hunting 0 2024-09-15T03:58:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.095+0000 7f0e0d0b9640 1 monclient: found mon.noname-c 2024-09-15T03:58:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.095+0000 7f0e0d0b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.095+0000 7f0e0d0b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.095+0000 7f0e0d0b9640 1 -- 172.21.15.110:0/1733398258 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e04009450 con 0x7f0e0809f180 2024-09-15T03:58:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.095+0000 7f0e0c8b8640 1 --2- 172.21.15.110:0/1733398258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e0809fa90 0x7f0e08119430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0e0d0b9640 1 -- 172.21.15.110:0/1733398258 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 msgr2=0x7f0e0812fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0e0d0b9640 1 --2- 172.21.15.110:0/1733398258 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 0x7f0e0812fb40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009970 tx=0x7f0e04033000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0dfd7fa640 1 -- 172.21.15.110:0/1733398258 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04033930 con 0x7f0e0809f180 2024-09-15T03:58:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0dfd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0dfd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0dfd7fa640 10 monclient: dump: 2024-09-15T03:58:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.108 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.109 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0dfd7fa640 10 monclient: _finish_auth 0 2024-09-15T03:58:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0dfd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0dfd7fa640 1 -- 172.21.15.110:0/1733398258 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e04033a90 con 0x7f0e0809f180 2024-09-15T03:58:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0e0f31d640 5 monclient: authenticate success, global_id 4304 2024-09-15T03:58:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0e0f31d640 10 monclient: _renew_subs 2024-09-15T03:58:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0e0f31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0e0f31d640 1 -- 172.21.15.110:0/1733398258 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e08099c80 con 0x7f0e0809f180 2024-09-15T03:58:48.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0dfd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:48.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0dfd7fa640 1 -- 172.21.15.110:0/1733398258 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e04033e30 con 0x7f0e0809f180 2024-09-15T03:58:48.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0dfd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0dfd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:48.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0dfd7fa640 10 monclient: dump: 2024-09-15T03:58:48.110 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.110 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.110 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.110 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.110 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.110 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.110 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.110 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.110 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.111 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0dff7fe640 4 set_mon_vals no callback set 2024-09-15T03:58:48.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0e0f31d640 10 monclient: _renew_subs 2024-09-15T03:58:48.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0e0f31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.097+0000 7f0e0f31d640 1 -- 172.21.15.110:0/1733398258 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0812efe0 con 0x7f0e0809f180 2024-09-15T03:58:48.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.098+0000 7f0e0f31d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:48.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.098+0000 7f0e0f31d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:48.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.098+0000 7f0e0f31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.098+0000 7f0e0f31d640 1 -- 172.21.15.110:0/1733398258 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0812efe0 con 0x7f0e0809f180 2024-09-15T03:58:48.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.098+0000 7f0dfd7fa640 1 -- 172.21.15.110:0/1733398258 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f0e0403a070 con 0x7f0e0809f180 2024-09-15T03:58:48.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.098+0000 7f0dfd7fa640 1 --2- 172.21.15.110:0/1733398258 >> v2:172.21.15.177:6812/28201 conn(0x7f0dec038700 0x7f0dec03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.098+0000 7f0dfd7fa640 1 -- 172.21.15.110:0/1733398258 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f0e0406bfe0 con 0x7f0e0809f180 2024-09-15T03:58:48.111 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.098+0000 7f0e0c8b8640 1 --2- 172.21.15.110:0/1733398258 >> v2:172.21.15.177:6812/28201 conn(0x7f0dec038700 0x7f0dec03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.098+0000 7f0e0c8b8640 10 monclient: get_auth_request con 0x7f0dec038700 auth_method 0 2024-09-15T03:58:48.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.101+0000 7f0e0c8b8640 1 --2- 172.21.15.110:0/1733398258 >> v2:172.21.15.177:6812/28201 conn(0x7f0dec038700 0x7f0dec03abb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0df8007fd0 tx=0x7f0df8007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:48.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.106+0000 7f0dfd7fa640 1 -- 172.21.15.110:0/1733398258 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f0e04048460 con 0x7f0e0809f180 2024-09-15T03:58:48.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.106+0000 7f0dfd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:48.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.106+0000 7f0dfd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:48.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.136+0000 7f612035d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:48.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.136+0000 7f612035d640 10 monclient: build_initial_monmap 2024-09-15T03:58:48.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.136+0000 7f612035d640 10 monclient: monmap: 2024-09-15T03:58:48.137 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:48.137 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.137 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:48.137535+0000 2024-09-15T03:58:48.138 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:48.137535+0000 2024-09-15T03:58:48.138 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:48.138 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.138 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:48.138 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:48.138 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:48.138 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:48.138 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:48.138 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 1 Processor -- start 2024-09-15T03:58:48.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 1 -- start start 2024-09-15T03:58:48.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 10 monclient: init 2024-09-15T03:58:48.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:48.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T03:58:48.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61180a1a10 0x7f6118131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 10 monclient(hunting): picked mon.noname-b con 0x7f61180a1a10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:48.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f611809e770 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:48.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61181369e0 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:48.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f611809f290 con 0x7f611809e770 2024-09-15T03:58:48.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61180a0850 con 0x7f61181369e0 2024-09-15T03:58:48.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f611809c9a0 con 0x7f61180a1a10 2024-09-15T03:58:48.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:48.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f612035d640 10 monclient(hunting): authenticate will time out at 751.781865s 2024-09-15T03:58:48.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f611e0f9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61180a1a10 0x7f6118131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f611d8f8640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f611809e770 0x7f611809eb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33866/0 (socket says 172.21.15.110:33866) 2024-09-15T03:58:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f611d8f8640 1 -- 172.21.15.110:0/468848783 learned_addr learned my addr 172.21.15.110:0/468848783 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f611e0f9640 10 monclient(hunting): get_auth_request con 0x7f61180a1a10 auth_method 0 2024-09-15T03:58:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f611e0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f611e0f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.137+0000 7f611e0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.138+0000 7f611e0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.138+0000 7f611e0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.138+0000 7f611e0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.138+0000 7f611e0f9640 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2024-09-15T03:58:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.138+0000 7f611e0f9640 1 -- 172.21.15.110:0/468848783 >> v1:172.21.15.177:6789/0 conn(0x7f61181369e0 legacy=0x7f6118138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.138+0000 7f611e0f9640 1 -- 172.21.15.110:0/468848783 >> v1:172.21.15.110:6789/0 conn(0x7f611809e770 legacy=0x7f611809eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.138+0000 7f611e0f9640 10 monclient: _finish_hunting 0 2024-09-15T03:58:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.138+0000 7f611e0f9640 1 monclient: found mon.noname-b 2024-09-15T03:58:48.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.138+0000 7f611e0f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:48.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.138+0000 7f611e0f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.138+0000 7f611e0f9640 1 -- 172.21.15.110:0/468848783 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611809cc80 con 0x7f61180a1a10 2024-09-15T03:58:48.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.140+0000 7f611e0f9640 1 -- 172.21.15.110:0/468848783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61180a1a10 msgr2=0x7f6118131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.140+0000 7f611e0f9640 1 --2- 172.21.15.110:0/468848783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61180a1a10 0x7f6118131b90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6114009880 tx=0x7f61140090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93c1f05ef4127b91 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f611d0f7640 1 -- 172.21.15.110:0/468848783 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6114032440 con 0x7f61180a1a10 2024-09-15T03:58:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f611d0f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f611d0f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f611d0f7640 10 monclient: dump: 2024-09-15T03:58:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.149 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.150 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f611d0f7640 10 monclient: _finish_auth 0 2024-09-15T03:58:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f611d0f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f612035d640 5 monclient: authenticate success, global_id 4307 2024-09-15T03:58:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f612035d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f611d0f7640 1 -- 172.21.15.110:0/468848783 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61140325a0 con 0x7f61180a1a10 2024-09-15T03:58:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f611d0f7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f612035d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f612035d640 4 set_mon_vals no callback set 2024-09-15T03:58:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f611d0f7640 1 -- 172.21.15.110:0/468848783 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6114032940 con 0x7f61180a1a10 2024-09-15T03:58:48.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f611d0f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f611d0f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:48.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f611d0f7640 10 monclient: dump: 2024-09-15T03:58:48.151 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.151 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.151 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.151 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.151 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.151 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.151 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.151 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.151 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.151 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f612035d640 10 monclient: shutdown 2024-09-15T03:58:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f612035d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f612035d640 1 -- 172.21.15.110:0/468848783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61180a1a10 msgr2=0x7f6118131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f612035d640 1 --2- 172.21.15.110:0/468848783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61180a1a10 0x7f6118131b90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6114009880 tx=0x7f61140090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f612035d640 1 -- 172.21.15.110:0/468848783 shutdown_connections 2024-09-15T03:58:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f612035d640 1 --2- 172.21.15.110:0/468848783 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f61180a1a10 0x7f6118131b90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f612035d640 1 -- 172.21.15.110:0/468848783 >> 172.21.15.110:0/468848783 conn(0x7f611806e860 msgr2=0x7f611806ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f612035d640 1 -- 172.21.15.110:0/468848783 shutdown_connections 2024-09-15T03:58:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.141+0000 7f612035d640 1 -- 172.21.15.110:0/468848783 wait complete. 2024-09-15T03:58:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 10 monclient: build_initial_monmap 2024-09-15T03:58:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 10 monclient: monmap: 2024-09-15T03:58:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:48.152 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:48.153 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:48.153 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:48.153 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.153 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:48.153 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:48.153 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:48.153 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 1 Processor -- start 2024-09-15T03:58:48.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 1 -- start start 2024-09-15T03:58:48.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 10 monclient: init 2024-09-15T03:58:48.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:48.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:58:48.153 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f611809e770 0x7f611809a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 10 monclient(hunting): picked mon.noname-c con 0x7f611809e770 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:48.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180a1a10 0x7f6118096200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 10 monclient(hunting): picked mon.noname-b con 0x7f61180a1a10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:48.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61181369e0 0x7f6118096740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 10 monclient(hunting): picked mon.noname-a con 0x7f61181369e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:48.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61180976e0 con 0x7f61181369e0 2024-09-15T03:58:48.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61180a11c0 con 0x7f61180a1a10 2024-09-15T03:58:48.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61180a3ea0 con 0x7f611809e770 2024-09-15T03:58:48.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:48.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.142+0000 7f612035d640 10 monclient(hunting): authenticate will time out at 751.786969s 2024-09-15T03:58:48.154 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e0f9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f611809e770 0x7f611809a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61181369e0 0x7f6118096740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61181369e0 0x7f6118096740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56668/0 (socket says 172.21.15.110:56668) 2024-09-15T03:58:48.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 1 -- 172.21.15.110:0/2064671540 learned_addr learned my addr 172.21.15.110:0/2064671540 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:48.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e0f9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f611809e770 0x7f611809a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55184/0 (socket says 172.21.15.110:55184) 2024-09-15T03:58:48.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 10 monclient(hunting): get_auth_request con 0x7f61181369e0 auth_method 0 2024-09-15T03:58:48.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e0f9640 10 monclient(hunting): get_auth_request con 0x7f611809e770 auth_method 0 2024-09-15T03:58:48.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e0f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e0f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e0f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.155 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e0f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e0f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e0f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611d8f8640 1 --2- 172.21.15.110:0/2064671540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180a1a10 0x7f6118096200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611d8f8640 10 monclient(hunting): get_auth_request con 0x7f61180a1a10 auth_method 0 2024-09-15T03:58:48.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611d8f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611d8f8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611d8f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611d8f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611d8f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.156 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611d8f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2024-09-15T03:58:48.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 1 -- 172.21.15.110:0/2064671540 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f611809e770 msgr2=0x7f611809a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 1 --2- 172.21.15.110:0/2064671540 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f611809e770 0x7f611809a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 1 -- 172.21.15.110:0/2064671540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180a1a10 msgr2=0x7f6118096200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 1 --2- 172.21.15.110:0/2064671540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180a1a10 0x7f6118096200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 10 monclient: _finish_hunting 0 2024-09-15T03:58:48.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 1 monclient: found mon.noname-a 2024-09-15T03:58:48.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:48.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:48.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e8fa640 1 -- 172.21.15.110:0/2064671540 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6114009390 con 0x7f61181369e0 2024-09-15T03:58:48.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.143+0000 7f611e0f9640 1 --2- 172.21.15.110:0/2064671540 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f611809e770 0x7f611809a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:48.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f611e8fa640 1 -- 172.21.15.110:0/2064671540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61181369e0 msgr2=0x7f6118096740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:48.157 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f611e8fa640 1 --2- 172.21.15.110:0/2064671540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61181369e0 0x7f6118096740 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f611000c890 tx=0x7f6110002640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f610e7fc640 1 -- 172.21.15.110:0/2064671540 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611000ce80 con 0x7f61181369e0 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f610e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f610e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f610e7fc640 10 monclient: dump: 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.158 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f610e7fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:48.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f610e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:48.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f612035d640 5 monclient: authenticate success, global_id 4293 2024-09-15T03:58:48.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f610e7fc640 1 -- 172.21.15.110:0/2064671540 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611000aa00 con 0x7f61181369e0 2024-09-15T03:58:48.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f610e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:48.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f610e7fc640 1 -- 172.21.15.110:0/2064671540 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611000ae00 con 0x7f61181369e0 2024-09-15T03:58:48.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f610e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f610e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:48.159 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f610e7fc640 10 monclient: dump: 2024-09-15T03:58:48.159 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.159 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.159 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.159 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.159 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.159 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.160 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.160 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.160 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.160 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f611c8f6640 4 set_mon_vals no callback set 2024-09-15T03:58:48.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f612035d640 10 monclient: _renew_subs 2024-09-15T03:58:48.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f612035d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:48.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f612035d640 1 -- 172.21.15.110:0/2064671540 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611809ad80 con 0x7f61181369e0 2024-09-15T03:58:48.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f612035d640 10 monclient: _renew_subs 2024-09-15T03:58:48.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f612035d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:48.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.145+0000 7f612035d640 1 -- 172.21.15.110:0/2064671540 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6118099850 con 0x7f61181369e0 2024-09-15T03:58:48.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.146+0000 7f612035d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:48.160 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.146+0000 7f612035d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:48.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.146+0000 7f612035d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:48.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.146+0000 7f612035d640 1 -- 172.21.15.110:0/2064671540 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6118099d70 con 0x7f61181369e0 2024-09-15T03:58:48.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.147+0000 7f610e7fc640 1 -- 172.21.15.110:0/2064671540 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f6110012630 con 0x7f61181369e0 2024-09-15T03:58:48.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.147+0000 7f610e7fc640 1 --2- 172.21.15.110:0/2064671540 >> v2:172.21.15.177:6812/28201 conn(0x7f60fc038700 0x7f60fc03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.147+0000 7f610e7fc640 1 -- 172.21.15.110:0/2064671540 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f611004b280 con 0x7f61181369e0 2024-09-15T03:58:48.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.147+0000 7f611e0f9640 1 --2- 172.21.15.110:0/2064671540 >> v2:172.21.15.177:6812/28201 conn(0x7f60fc038700 0x7f60fc03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.147+0000 7f611e0f9640 10 monclient: get_auth_request con 0x7f60fc038700 auth_method 0 2024-09-15T03:58:48.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.150+0000 7f611e0f9640 1 --2- 172.21.15.110:0/2064671540 >> v2:172.21.15.177:6812/28201 conn(0x7f60fc038700 0x7f60fc03abb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6118069a30 tx=0x7f6114032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:48.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.151+0000 7f610e7fc640 1 -- 172.21.15.110:0/2064671540 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6118099d70 con 0x7f61181369e0 2024-09-15T03:58:48.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.151+0000 7f610e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:48.162 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.151+0000 7f610e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.270+0000 7f0e0f31d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:58:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.270+0000 7f0e0f31d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:58:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.270+0000 7f0e0f31d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.271+0000 7f0e0f31d640 1 -- 172.21.15.110:0/1733398258 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0e0812efe0 con 0x7f0e0809f180 2024-09-15T03:58:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.271+0000 7f0dfd7fa640 1 -- 172.21.15.110:0/1733398258 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0e04038030 con 0x7f0e0809f180 2024-09-15T03:58:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.271+0000 7f0dfd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:58:48.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.271+0000 7f0dfd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:48.272 INFO:teuthology.orchestra.run.smithi110.stdout:47244640260 2024-09-15T03:58:48.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.273+0000 7f0e0f31d640 1 -- 172.21.15.110:0/1733398258 >> v2:172.21.15.177:6812/28201 conn(0x7f0dec038700 msgr2=0x7f0dec03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.273+0000 7f0e0f31d640 1 --2- 172.21.15.110:0/1733398258 >> v2:172.21.15.177:6812/28201 conn(0x7f0dec038700 0x7f0dec03abb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0df8007fd0 tx=0x7f0df8007e10 comp rx=0 tx=0).stop 2024-09-15T03:58:48.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.273+0000 7f0e0f31d640 10 monclient: shutdown 2024-09-15T03:58:48.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.273+0000 7f0e0f31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:48.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.273+0000 7f0e0f31d640 1 -- 172.21.15.110:0/1733398258 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 msgr2=0x7f0e0812fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.273+0000 7f0e0f31d640 1 --2- 172.21.15.110:0/1733398258 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 0x7f0e0812fb40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009970 tx=0x7f0e04033000 comp rx=0 tx=0).stop 2024-09-15T03:58:48.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.273+0000 7f0e0f31d640 1 -- 172.21.15.110:0/1733398258 shutdown_connections 2024-09-15T03:58:48.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.273+0000 7f0e0f31d640 1 --2- 172.21.15.110:0/1733398258 >> v2:172.21.15.177:6812/28201 conn(0x7f0dec038700 0x7f0dec03abb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.273+0000 7f0e0f31d640 1 --2- 172.21.15.110:0/1733398258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e081331f0 0x7f0e08119970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.273+0000 7f0e0f31d640 1 --2- 172.21.15.110:0/1733398258 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0e0809fa90 0x7f0e08119430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.273+0000 7f0e0f31d640 1 --2- 172.21.15.110:0/1733398258 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0e0809f180 0x7f0e0812fb40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.273+0000 7f0e0f31d640 1 -- 172.21.15.110:0/1733398258 >> 172.21.15.110:0/1733398258 conn(0x7f0e0809d680 msgr2=0x7f0e08099fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:48.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.273+0000 7f0e0f31d640 1 -- 172.21.15.110:0/1733398258 shutdown_connections 2024-09-15T03:58:48.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.273+0000 7f0e0f31d640 1 -- 172.21.15.110:0/1733398258 wait complete. 2024-09-15T03:58:48.284 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.5 2024-09-15T03:58:48.284 DEBUG:teuthology.parallel:result is None 2024-09-15T03:58:48.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.311+0000 7f612035d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:58:48.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.311+0000 7f612035d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:58:48.312 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.311+0000 7f612035d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:48.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.311+0000 7f612035d640 1 -- 172.21.15.110:0/2064671540 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f6118099d70 con 0x7f61181369e0 2024-09-15T03:58:48.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.312+0000 7f610e7fc640 1 -- 172.21.15.110:0/2064671540 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6118099d70 con 0x7f61181369e0 2024-09-15T03:58:48.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.312+0000 7f610e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:58:48.313 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.312+0000 7f610e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:48.313 INFO:teuthology.orchestra.run.smithi110.stdout:47244640260 2024-09-15T03:58:48.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.314+0000 7f612035d640 1 -- 172.21.15.110:0/2064671540 >> v2:172.21.15.177:6812/28201 conn(0x7f60fc038700 msgr2=0x7f60fc03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.314+0000 7f612035d640 1 --2- 172.21.15.110:0/2064671540 >> v2:172.21.15.177:6812/28201 conn(0x7f60fc038700 0x7f60fc03abb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6118069a30 tx=0x7f6114032000 comp rx=0 tx=0).stop 2024-09-15T03:58:48.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.314+0000 7f612035d640 10 monclient: shutdown 2024-09-15T03:58:48.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.314+0000 7f612035d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:48.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.314+0000 7f612035d640 1 -- 172.21.15.110:0/2064671540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61181369e0 msgr2=0x7f6118096740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.314+0000 7f612035d640 1 --2- 172.21.15.110:0/2064671540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61181369e0 0x7f6118096740 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f611000c890 tx=0x7f6110002640 comp rx=0 tx=0).stop 2024-09-15T03:58:48.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.314+0000 7f612035d640 1 -- 172.21.15.110:0/2064671540 shutdown_connections 2024-09-15T03:58:48.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.314+0000 7f612035d640 1 --2- 172.21.15.110:0/2064671540 >> v2:172.21.15.177:6812/28201 conn(0x7f60fc038700 0x7f60fc03abb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.314+0000 7f612035d640 1 --2- 172.21.15.110:0/2064671540 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f61181369e0 0x7f6118096740 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.314+0000 7f612035d640 1 --2- 172.21.15.110:0/2064671540 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f61180a1a10 0x7f6118096200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.314+0000 7f612035d640 1 --2- 172.21.15.110:0/2064671540 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f611809e770 0x7f611809a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.314+0000 7f612035d640 1 -- 172.21.15.110:0/2064671540 >> 172.21.15.110:0/2064671540 conn(0x7f611806e860 msgr2=0x7f6118138dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:48.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.314+0000 7f612035d640 1 -- 172.21.15.110:0/2064671540 shutdown_connections 2024-09-15T03:58:48.316 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.314+0000 7f612035d640 1 -- 172.21.15.110:0/2064671540 wait complete. 2024-09-15T03:58:48.325 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640260 for osd.3 2024-09-15T03:58:48.325 DEBUG:teuthology.parallel:result is None 2024-09-15T03:58:48.325 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T03:58:48.325 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:58:48.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.456+0000 7f590755d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:48.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.456+0000 7f590755d640 10 monclient: build_initial_monmap 2024-09-15T03:58:48.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.456+0000 7f590755d640 10 monclient: monmap: 2024-09-15T03:58:48.457 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:48.457 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.457 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:48.457498+0000 2024-09-15T03:58:48.458 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:48.457498+0000 2024-09-15T03:58:48.458 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:48.458 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.458 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:48.458 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:48.458 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:48.458 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:48.458 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:48.458 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 1 Processor -- start 2024-09-15T03:58:48.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 1 -- start start 2024-09-15T03:58:48.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 10 monclient: init 2024-09-15T03:58:48.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:48.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-15T03:58:48.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f59000a1bf0 0x7f5900131810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 10 monclient(hunting): picked mon.noname-c con 0x7f59000a1bf0 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:48.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f590009e780 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:48.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5900136a40 0x7f5900138e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 10 monclient(hunting): picked mon.noname-b con 0x7f5900136a40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:48.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f590009c9b0 con 0x7f590009e780 2024-09-15T03:58:48.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59000a0c20 con 0x7f59000a1bf0 2024-09-15T03:58:48.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590009cc90 con 0x7f5900136a40 2024-09-15T03:58:48.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:48.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f590755d640 10 monclient(hunting): authenticate will time out at 752.101745s 2024-09-15T03:58:48.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f5905afa640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5900136a40 0x7f5900138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f5904af8640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f590009e780 0x7f590009eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33878/0 (socket says 172.21.15.110:33878) 2024-09-15T03:58:48.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f5904af8640 1 -- 172.21.15.110:0/1475778017 learned_addr learned my addr 172.21.15.110:0/1475778017 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:48.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f5905afa640 10 monclient(hunting): get_auth_request con 0x7f5900136a40 auth_method 0 2024-09-15T03:58:48.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f5905afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f5905afa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.457+0000 7f5905afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.458+0000 7f5905afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.458+0000 7f5905afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.458+0000 7f5905afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.458+0000 7f59052f9640 1 --2- 172.21.15.110:0/1475778017 >> v2:172.21.15.177:3300/0 conn(0x7f59000a1bf0 0x7f5900131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.458+0000 7f5905afa640 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2024-09-15T03:58:48.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.458+0000 7f5905afa640 1 -- 172.21.15.110:0/1475778017 >> v2:172.21.15.177:3300/0 conn(0x7f59000a1bf0 msgr2=0x7f5900131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:48.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.458+0000 7f5905afa640 1 --2- 172.21.15.110:0/1475778017 >> v2:172.21.15.177:3300/0 conn(0x7f59000a1bf0 0x7f5900131810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.458+0000 7f5905afa640 1 -- 172.21.15.110:0/1475778017 >> v1:172.21.15.110:6789/0 conn(0x7f590009e780 legacy=0x7f590009eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.458+0000 7f5905afa640 10 monclient: _finish_hunting 0 2024-09-15T03:58:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.458+0000 7f5905afa640 1 monclient: found mon.noname-b 2024-09-15T03:58:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.458+0000 7f5905afa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.458+0000 7f5905afa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.463 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.458+0000 7f5905afa640 1 -- 172.21.15.110:0/1475778017 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59000a1fc0 con 0x7f5900136a40 2024-09-15T03:58:48.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.460+0000 7f5905afa640 1 -- 172.21.15.110:0/1475778017 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5900136a40 msgr2=0x7f5900138e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:48.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.460+0000 7f5905afa640 1 --2- 172.21.15.110:0/1475778017 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5900136a40 0x7f5900138e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5900072140 tx=0x7f58f8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a84e1e2aa00d179 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:48.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f58f7fff640 1 -- 172.21.15.110:0/1475778017 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f8007630 con 0x7f5900136a40 2024-09-15T03:58:48.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f58f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f58f7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:48.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f58f7fff640 10 monclient: dump: 2024-09-15T03:58:48.469 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.469 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.469 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.469 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.469 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.470 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.470 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.470 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.470 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.470 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f58f7fff640 10 monclient: _finish_auth 0 2024-09-15T03:58:48.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f58f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:48.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f58f7fff640 1 -- 172.21.15.110:0/1475778017 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f802d050 con 0x7f5900136a40 2024-09-15T03:58:48.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f590755d640 5 monclient: authenticate success, global_id 4313 2024-09-15T03:58:48.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f590755d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:48.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f58f7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:48.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f590755d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:48.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f58f7fff640 1 -- 172.21.15.110:0/1475778017 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f8007a00 con 0x7f5900136a40 2024-09-15T03:58:48.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f590755d640 4 set_mon_vals no callback set 2024-09-15T03:58:48.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f58f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f58f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:48.471 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f58f7fff640 10 monclient: dump: 2024-09-15T03:58:48.471 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.471 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.471 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.471 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.471 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.471 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.471 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.471 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.471 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.471 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f590755d640 10 monclient: shutdown 2024-09-15T03:58:48.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f590755d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:48.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f590755d640 1 -- 172.21.15.110:0/1475778017 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5900136a40 msgr2=0x7f5900138e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f590755d640 1 --2- 172.21.15.110:0/1475778017 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5900136a40 0x7f5900138e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5900072140 tx=0x7f58f8009780 comp rx=0 tx=0).stop 2024-09-15T03:58:48.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f590755d640 1 -- 172.21.15.110:0/1475778017 shutdown_connections 2024-09-15T03:58:48.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f590755d640 1 --2- 172.21.15.110:0/1475778017 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5900136a40 0x7f5900138e20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f590755d640 1 --2- 172.21.15.110:0/1475778017 >> v2:172.21.15.177:3300/0 conn(0x7f59000a1bf0 0x7f5900131810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f590755d640 1 -- 172.21.15.110:0/1475778017 >> 172.21.15.110:0/1475778017 conn(0x7f5900062dd0 msgr2=0x7f59000631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:48.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f590755d640 1 -- 172.21.15.110:0/1475778017 shutdown_connections 2024-09-15T03:58:48.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.461+0000 7f590755d640 1 -- 172.21.15.110:0/1475778017 wait complete. 2024-09-15T03:58:48.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.462+0000 7f590755d640 10 monclient: build_initial_monmap 2024-09-15T03:58:48.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.462+0000 7f590755d640 10 monclient: monmap: 2024-09-15T03:58:48.472 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:48.472 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:48.473 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:48.473 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:48.473 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:48.473 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.473 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:48.473 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:48.473 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:48.473 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.462+0000 7f590755d640 1 Processor -- start 2024-09-15T03:58:48.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.462+0000 7f590755d640 1 -- start start 2024-09-15T03:58:48.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.462+0000 7f590755d640 10 monclient: init 2024-09-15T03:58:48.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:48.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:58:48.473 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f590009e780 0x7f5900130860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 10 monclient(hunting): picked mon.noname-b con 0x7f590009e780 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59000a1bf0 0x7f5900099ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 10 monclient(hunting): picked mon.noname-c con 0x7f59000a1bf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5900136a40 0x7f590009a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 10 monclient(hunting): picked mon.noname-a con 0x7f5900136a40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590011cc50 con 0x7f5900136a40 2024-09-15T03:58:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59000a2ea0 con 0x7f590009e780 2024-09-15T03:58:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590009c250 con 0x7f59000a1bf0 2024-09-15T03:58:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f590755d640 10 monclient(hunting): authenticate will time out at 752.107169s 2024-09-15T03:58:48.474 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59000a1bf0 0x7f5900099ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5905afa640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5900136a40 0x7f590009a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5905afa640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5900136a40 0x7f590009a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56672/0 (socket says 172.21.15.110:56672) 2024-09-15T03:58:48.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59000a1bf0 0x7f5900099ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55194/0 (socket says 172.21.15.110:55194) 2024-09-15T03:58:48.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5905afa640 1 -- 172.21.15.110:0/2048905302 learned_addr learned my addr 172.21.15.110:0/2048905302 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:48.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5905afa640 10 monclient(hunting): get_auth_request con 0x7f5900136a40 auth_method 0 2024-09-15T03:58:48.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5905afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5905afa640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5905afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 10 monclient(hunting): get_auth_request con 0x7f59000a1bf0 auth_method 0 2024-09-15T03:58:48.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.475 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5905afa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5905afa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f59052f9640 1 --2- 172.21.15.110:0/2048905302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f590009e780 0x7f5900130860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5905afa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f59052f9640 10 monclient(hunting): get_auth_request con 0x7f590009e780 auth_method 0 2024-09-15T03:58:48.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f59052f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f59052f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f59052f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f59052f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f59052f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f59052f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-09-15T03:58:48.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 1 -- 172.21.15.110:0/2048905302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f590009e780 msgr2=0x7f5900130860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 1 --2- 172.21.15.110:0/2048905302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f590009e780 0x7f5900130860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 1 -- 172.21.15.110:0/2048905302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5900136a40 msgr2=0x7f590009a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 1 --2- 172.21.15.110:0/2048905302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5900136a40 0x7f590009a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 10 monclient: _finish_hunting 0 2024-09-15T03:58:48.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 1 monclient: found mon.noname-c 2024-09-15T03:58:48.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:48.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5904af8640 1 -- 172.21.15.110:0/2048905302 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f8009a20 con 0x7f59000a1bf0 2024-09-15T03:58:48.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.463+0000 7f5905afa640 1 --2- 172.21.15.110:0/2048905302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5900136a40 0x7f590009a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:48.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f5904af8640 1 -- 172.21.15.110:0/2048905302 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59000a1bf0 msgr2=0x7f5900099ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:48.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f5904af8640 1 --2- 172.21.15.110:0/2048905302 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59000a1bf0 0x7f5900099ca0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f58f000c490 tx=0x7f58f0007f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:48.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f58f57fa640 1 -- 172.21.15.110:0/2048905302 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0005cb0 con 0x7f59000a1bf0 2024-09-15T03:58:48.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f58f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f58f57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:48.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f58f57fa640 10 monclient: dump: 2024-09-15T03:58:48.478 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.478 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.478 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.478 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.478 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.478 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.478 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.478 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.478 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.478 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f58f57fa640 10 monclient: _finish_auth 0 2024-09-15T03:58:48.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f58f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:48.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f590755d640 5 monclient: authenticate success, global_id 4316 2024-09-15T03:58:48.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f58f57fa640 1 -- 172.21.15.110:0/2048905302 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f0005e30 con 0x7f59000a1bf0 2024-09-15T03:58:48.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f590755d640 10 monclient: _renew_subs 2024-09-15T03:58:48.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f590755d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f590755d640 1 -- 172.21.15.110:0/2048905302 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f590011c970 con 0x7f59000a1bf0 2024-09-15T03:58:48.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f58f57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:48.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f58f57fa640 1 -- 172.21.15.110:0/2048905302 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f0007320 con 0x7f59000a1bf0 2024-09-15T03:58:48.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f58f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.466+0000 7f58f57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:48.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.466+0000 7f58f57fa640 10 monclient: dump: 2024-09-15T03:58:48.479 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.480 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.480 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.480 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.480 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.480 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.480 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.480 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.480 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.480 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.466+0000 7f590755d640 10 monclient: _renew_subs 2024-09-15T03:58:48.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.465+0000 7f58f77fe640 4 set_mon_vals no callback set 2024-09-15T03:58:48.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.466+0000 7f590755d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.466+0000 7f590755d640 1 -- 172.21.15.110:0/2048905302 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590009b750 con 0x7f59000a1bf0 2024-09-15T03:58:48.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.466+0000 7f58f57fa640 1 -- 172.21.15.110:0/2048905302 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f58f001b3b0 con 0x7f59000a1bf0 2024-09-15T03:58:48.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.466+0000 7f590755d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:48.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.466+0000 7f590755d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:48.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.466+0000 7f590755d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.466+0000 7f590755d640 1 -- 172.21.15.110:0/2048905302 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5900070bd0 con 0x7f59000a1bf0 2024-09-15T03:58:48.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.466+0000 7f58f57fa640 1 --2- 172.21.15.110:0/2048905302 >> v2:172.21.15.177:6812/28201 conn(0x7f58e4038700 0x7f58e403abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.467+0000 7f58f57fa640 1 -- 172.21.15.110:0/2048905302 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f58f004aaa0 con 0x7f59000a1bf0 2024-09-15T03:58:48.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.467+0000 7f59052f9640 1 --2- 172.21.15.110:0/2048905302 >> v2:172.21.15.177:6812/28201 conn(0x7f58e4038700 0x7f58e403abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.467+0000 7f59052f9640 10 monclient: get_auth_request con 0x7f58e4038700 auth_method 0 2024-09-15T03:58:48.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.469+0000 7f59052f9640 1 --2- 172.21.15.110:0/2048905302 >> v2:172.21.15.177:6812/28201 conn(0x7f58e4038700 0x7f58e403abb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00ac20 tx=0x7f58fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:48.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.471+0000 7f58f57fa640 1 -- 172.21.15.110:0/2048905302 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5900070bd0 con 0x7f59000a1bf0 2024-09-15T03:58:48.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.471+0000 7f58f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:48.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.471+0000 7f58f57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:48.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.623+0000 7f590755d640 1 -- 172.21.15.110:0/2048905302 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5900070bd0 con 0x7f58e4038700 2024-09-15T03:58:48.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.625+0000 7f58f57fa640 1 -- 172.21.15.110:0/2048905302 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+39252 (secure 0 0 0) 0x7f5900070bd0 con 0x7f58e4038700 2024-09-15T03:58:48.627 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:48.627 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-09-15T03:58:48.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.627+0000 7f590755d640 1 -- 172.21.15.110:0/2048905302 >> v2:172.21.15.177:6812/28201 conn(0x7f58e4038700 msgr2=0x7f58e403abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.627+0000 7f590755d640 1 --2- 172.21.15.110:0/2048905302 >> v2:172.21.15.177:6812/28201 conn(0x7f58e4038700 0x7f58e403abb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00ac20 tx=0x7f58fc008040 comp rx=0 tx=0).stop 2024-09-15T03:58:48.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.627+0000 7f590755d640 10 monclient: shutdown 2024-09-15T03:58:48.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.627+0000 7f590755d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:48.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.627+0000 7f590755d640 1 -- 172.21.15.110:0/2048905302 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59000a1bf0 msgr2=0x7f5900099ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.627+0000 7f590755d640 1 --2- 172.21.15.110:0/2048905302 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59000a1bf0 0x7f5900099ca0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f58f000c490 tx=0x7f58f0007f00 comp rx=0 tx=0).stop 2024-09-15T03:58:48.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.627+0000 7f590755d640 1 -- 172.21.15.110:0/2048905302 shutdown_connections 2024-09-15T03:58:48.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.627+0000 7f590755d640 1 --2- 172.21.15.110:0/2048905302 >> v2:172.21.15.177:6812/28201 conn(0x7f58e4038700 0x7f58e403abb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.627+0000 7f590755d640 1 --2- 172.21.15.110:0/2048905302 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5900136a40 0x7f590009a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.627+0000 7f590755d640 1 --2- 172.21.15.110:0/2048905302 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f59000a1bf0 0x7f5900099ca0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.627+0000 7f590755d640 1 --2- 172.21.15.110:0/2048905302 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f590009e780 0x7f5900130860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.627+0000 7f590755d640 1 -- 172.21.15.110:0/2048905302 >> 172.21.15.110:0/2048905302 conn(0x7f5900062dd0 msgr2=0x7f59000a30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:48.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.627+0000 7f590755d640 1 -- 172.21.15.110:0/2048905302 shutdown_connections 2024-09-15T03:58:48.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.627+0000 7f590755d640 1 -- 172.21.15.110:0/2048905302 wait complete. 2024-09-15T03:58:48.638 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-15T03:58:48.363161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562077696,"kb_used":4777392,"kb_used_data":557300568,"kb_used_omap":443,"kb_used_meta":0,"kb_avail":557300304,"statfs":{"total":575567560704,"available":570675511296,"internally_reserved":270336,"allocated":570675781632,"data_stored":570675781632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":453772,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001072"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.072207+0000","last_change":"2024-09-15T03:58:44.072431+0000","last_active":"2024-09-15T03:58:44.072207+0000","last_peered":"2024-09-15T03:58:44.072207+0000","last_clean":"2024-09-15T03:58:44.072207+0000","last_became_active":"2024-09-15T03:58:42.157123+0000","last_became_peered":"2024-09-15T03:58:42.157123+0000","last_unstale":"2024-09-15T03:58:44.072207+0000","last_undegraded":"2024-09-15T03:58:44.072207+0000","last_fullsized":"2024-09-15T03:58:44.072207+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:46:37.797943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066518800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071760+0000","last_change":"2024-09-15T03:58:44.072012+0000","last_active":"2024-09-15T03:58:44.071760+0000","last_peered":"2024-09-15T03:58:44.071760+0000","last_clean":"2024-09-15T03:58:44.071760+0000","last_became_active":"2024-09-15T03:58:42.069324+0000","last_became_peered":"2024-09-15T03:58:42.069324+0000","last_unstale":"2024-09-15T03:58:44.071760+0000","last_undegraded":"2024-09-15T03:58:44.071760+0000","last_fullsized":"2024-09-15T03:58:44.071760+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:44:04.033148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040535300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071964+0000","last_change":"2024-09-15T03:58:44.072166+0000","last_active":"2024-09-15T03:58:44.071964+0000","last_peered":"2024-09-15T03:58:44.071964+0000","last_clean":"2024-09-15T03:58:44.071964+0000","last_became_active":"2024-09-15T03:58:42.155521+0000","last_became_peered":"2024-09-15T03:58:42.155521+0000","last_unstale":"2024-09-15T03:58:44.071964+0000","last_undegraded":"2024-09-15T03:58:44.071964+0000","last_fullsized":"2024-09-15T03:58:44.071964+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:42:01.920640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040527199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071921+0000","last_change":"2024-09-15T03:58:44.072137+0000","last_active":"2024-09-15T03:58:44.071921+0000","last_peered":"2024-09-15T03:58:44.071921+0000","last_clean":"2024-09-15T03:58:44.071921+0000","last_became_active":"2024-09-15T03:58:42.155317+0000","last_became_peered":"2024-09-15T03:58:42.155317+0000","last_unstale":"2024-09-15T03:58:44.071921+0000","last_undegraded":"2024-09-15T03:58:44.071921+0000","last_fullsized":"2024-09-15T03:58:44.071921+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:55:28.163584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041321199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.074749+0000","last_change":"2024-09-15T03:58:44.074749+0000","last_active":"2024-09-15T03:58:44.074749+0000","last_peered":"2024-09-15T03:58:44.074749+0000","last_clean":"2024-09-15T03:58:44.074749+0000","last_became_active":"2024-09-15T03:58:42.071376+0000","last_became_peered":"2024-09-15T03:58:42.071376+0000","last_unstale":"2024-09-15T03:58:44.074749+0000","last_undegraded":"2024-09-15T03:58:44.074749+0000","last_fullsized":"2024-09-15T03:58:44.074749+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:42:52.640034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080084100000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071218+0000","last_change":"2024-09-15T03:58:44.071453+0000","last_active":"2024-09-15T03:58:44.071218+0000","last_peered":"2024-09-15T03:58:44.071218+0000","last_clean":"2024-09-15T03:58:44.071218+0000","last_became_active":"2024-09-15T03:58:42.067012+0000","last_became_peered":"2024-09-15T03:58:42.067012+0000","last_unstale":"2024-09-15T03:58:44.071218+0000","last_undegraded":"2024-09-15T03:58:44.071218+0000","last_fullsized":"2024-09-15T03:58:44.071218+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:04:08.843151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040766700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071926+0000","last_change":"2024-09-15T03:58:44.072155+0000","last_active":"2024-09-15T03:58:44.071926+0000","last_peered":"2024-09-15T03:58:44.071926+0000","last_clean":"2024-09-15T03:58:44.071926+0000","last_became_active":"2024-09-15T03:58:42.069300+0000","last_became_peered":"2024-09-15T03:58:42.069300+0000","last_unstale":"2024-09-15T03:58:44.071926+0000","last_undegraded":"2024-09-15T03:58:44.071926+0000","last_fullsized":"2024-09-15T03:58:44.071926+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:28:34.591535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036612400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.074041+0000","last_change":"2024-09-15T03:58:44.074233+0000","last_active":"2024-09-15T03:58:44.074041+0000","last_peered":"2024-09-15T03:58:44.074041+0000","last_clean":"2024-09-15T03:58:44.074041+0000","last_became_active":"2024-09-15T03:58:42.071652+0000","last_became_peered":"2024-09-15T03:58:42.071652+0000","last_unstale":"2024-09-15T03:58:44.074041+0000","last_undegraded":"2024-09-15T03:58:44.074041+0000","last_fullsized":"2024-09-15T03:58:44.074041+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:31:44.288660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030212199999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":103,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T03:58:43.157400+0000","last_change":"2024-09-15T03:58:38.057397+0000","last_active":"2024-09-15T03:58:43.157400+0000","last_peered":"2024-09-15T03:58:43.157400+0000","last_clean":"2024-09-15T03:58:43.157400+0000","last_became_active":"2024-09-15T03:58:38.056673+0000","last_became_peered":"2024-09-15T03:58:38.056673+0000","last_unstale":"2024-09-15T03:58:43.157400+0000","last_undegraded":"2024-09-15T03:58:43.157400+0000","last_fullsized":"2024-09-15T03:58:43.157400+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_clean_scrub_stamp":"2024-09-15T03:58:37.036429+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:38:58.792683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795816,"kb_used_data":92883832,"kb_used_omap":56,"kb_used_meta":0,"kb_avail":92883800,"statfs":{"total":95927926784,"available":95113011200,"internally_reserved":32768,"allocated":95113043968,"data_stored":95113043968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58360,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795844,"kb_used_data":92883816,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92883772,"statfs":{"total":95927926784,"available":95112982528,"internally_reserved":45056,"allocated":95113027584,"data_stored":95113027584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48544,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797028,"kb_used_data":92882636,"kb_used_omap":109,"kb_used_meta":0,"kb_avail":92882588,"statfs":{"total":95927926784,"available":95111770112,"internally_reserved":49152,"allocated":95111819264,"data_stored":95111819264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112614,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795784,"kb_used_data":92883856,"kb_used_omap":55,"kb_used_meta":0,"kb_avail":92883832,"statfs":{"total":95927926784,"available":95113043968,"internally_reserved":24576,"allocated":95113068544,"data_stored":95113068544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":56457,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795840,"kb_used_data":92883828,"kb_used_omap":74,"kb_used_meta":0,"kb_avail":92883776,"statfs":{"total":95927926784,"available":95112986624,"internally_reserved":53248,"allocated":95113039872,"data_stored":95113039872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76777,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797080,"kb_used_data":92882600,"kb_used_omap":98,"kb_used_meta":0,"kb_avail":92882536,"statfs":{"total":95927926784,"available":95111716864,"internally_reserved":65536,"allocated":95111782400,"data_stored":95111782400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101020,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]}]}],"pool_statfs":[]}} 2024-09-15T03:58:48.640 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:58:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.731+0000 7f6193954640 10 monclient: get_monmap_and_config 2024-09-15T03:58:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.731+0000 7f6193954640 10 monclient: build_initial_monmap 2024-09-15T03:58:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.731+0000 7f6193954640 10 monclient: monmap: 2024-09-15T03:58:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.732 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:48.732488+0000 2024-09-15T03:58:48.733 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:48.732488+0000 2024-09-15T03:58:48.733 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:48.733 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.733 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:48.733 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:48.733 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:48.733 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:48.733 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:48.733 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 1 Processor -- start 2024-09-15T03:58:48.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 1 -- start start 2024-09-15T03:58:48.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 10 monclient: init 2024-09-15T03:58:48.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:48.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-15T03:58:48.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f618c06e430 0x7f618c06a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 10 monclient(hunting): picked mon.noname-c con 0x7f618c06e430 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:48.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f618c06e800 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:48.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06ec20 0x7f618c12e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 10 monclient(hunting): picked mon.noname-b con 0x7f618c06ec20 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:48.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618c09b660 con 0x7f618c06e800 2024-09-15T03:58:48.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f618c06c940 con 0x7f618c06e430 2024-09-15T03:58:48.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f618c06cc20 con 0x7f618c06ec20 2024-09-15T03:58:48.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:48.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6193954640 10 monclient(hunting): authenticate will time out at 752.376762s 2024-09-15T03:58:48.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6191ef1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06ec20 0x7f618c12e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6191ef1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06ec20 0x7f618c12e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55200/0 (socket says 172.21.15.110:55200) 2024-09-15T03:58:48.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6191ef1640 1 -- 172.21.15.110:0/1441738595 learned_addr learned my addr 172.21.15.110:0/1441738595 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:48.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6191ef1640 10 monclient(hunting): get_auth_request con 0x7f618c06ec20 auth_method 0 2024-09-15T03:58:48.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6191ef1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6191ef1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.732+0000 7f6191ef1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f6191ef1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f6191ef1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f6191ef1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f61916f0640 1 --2- 172.21.15.110:0/1441738595 >> v2:172.21.15.177:3300/0 conn(0x7f618c06e430 0x7f618c06a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f61916f0640 10 monclient(hunting): get_auth_request con 0x7f618c06e430 auth_method 0 2024-09-15T03:58:48.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f61916f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f61916f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f61916f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f61916f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f61916f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f61916f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f6191ef1640 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2024-09-15T03:58:48.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f6191ef1640 1 -- 172.21.15.110:0/1441738595 >> v2:172.21.15.177:3300/0 conn(0x7f618c06e430 msgr2=0x7f618c06a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:48.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f6191ef1640 1 --2- 172.21.15.110:0/1441738595 >> v2:172.21.15.177:3300/0 conn(0x7f618c06e430 0x7f618c06a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f6191ef1640 1 -- 172.21.15.110:0/1441738595 >> v1:172.21.15.177:6789/0 conn(0x7f618c06e800 legacy=0x7f618c12e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:48.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f6191ef1640 10 monclient: _finish_hunting 0 2024-09-15T03:58:48.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f6191ef1640 1 monclient: found mon.noname-b 2024-09-15T03:58:48.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f6191ef1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:48.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f6191ef1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.733+0000 7f6191ef1640 1 -- 172.21.15.110:0/1441738595 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c06cf00 con 0x7f618c06ec20 2024-09-15T03:58:48.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.735+0000 7f6191ef1640 1 -- 172.21.15.110:0/1441738595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06ec20 msgr2=0x7f618c12e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:48.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.735+0000 7f6191ef1640 1 --2- 172.21.15.110:0/1441738595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06ec20 0x7f618c12e9b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6184009f20 tx=0x7f6184009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf073dd10cbd3643 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:48.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6183fff640 1 -- 172.21.15.110:0/1441738595 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6184007610 con 0x7f618c06ec20 2024-09-15T03:58:48.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6183fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6183fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:48.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6183fff640 10 monclient: dump: 2024-09-15T03:58:48.746 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.746 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.746 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.746 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.746 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.746 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.746 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.746 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.746 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.746 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6183fff640 10 monclient: _finish_auth 0 2024-09-15T03:58:48.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6183fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:48.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6183fff640 1 -- 172.21.15.110:0/1441738595 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618402d050 con 0x7f618c06ec20 2024-09-15T03:58:48.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6193954640 5 monclient: authenticate success, global_id 4319 2024-09-15T03:58:48.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6193954640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:48.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6183fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:48.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6183fff640 1 -- 172.21.15.110:0/1441738595 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61840079a0 con 0x7f618c06ec20 2024-09-15T03:58:48.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6193954640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:48.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6193954640 4 set_mon_vals no callback set 2024-09-15T03:58:48.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6183fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6183fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:48.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6183fff640 10 monclient: dump: 2024-09-15T03:58:48.747 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.747 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.748 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.748 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.748 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.748 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.748 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.748 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.748 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.748 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6193954640 10 monclient: shutdown 2024-09-15T03:58:48.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6193954640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:48.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6193954640 1 -- 172.21.15.110:0/1441738595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06ec20 msgr2=0x7f618c12e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6193954640 1 --2- 172.21.15.110:0/1441738595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06ec20 0x7f618c12e9b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6184009f20 tx=0x7f6184009840 comp rx=0 tx=0).stop 2024-09-15T03:58:48.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6193954640 1 -- 172.21.15.110:0/1441738595 shutdown_connections 2024-09-15T03:58:48.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6193954640 1 --2- 172.21.15.110:0/1441738595 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06ec20 0x7f618c12e9b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6193954640 1 --2- 172.21.15.110:0/1441738595 >> v2:172.21.15.177:3300/0 conn(0x7f618c06e430 0x7f618c06a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6193954640 1 -- 172.21.15.110:0/1441738595 >> 172.21.15.110:0/1441738595 conn(0x7f618c09d8f0 msgr2=0x7f618c09dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:48.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6193954640 1 -- 172.21.15.110:0/1441738595 shutdown_connections 2024-09-15T03:58:48.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.736+0000 7f6193954640 1 -- 172.21.15.110:0/1441738595 wait complete. 2024-09-15T03:58:48.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.737+0000 7f6193954640 10 monclient: build_initial_monmap 2024-09-15T03:58:48.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.737+0000 7f6193954640 10 monclient: monmap: 2024-09-15T03:58:48.749 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:48.749 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:48.749 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:48.750 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:48.750 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:48.750 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.750 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:48.750 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:48.750 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:48.750 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.737+0000 7f6193954640 1 Processor -- start 2024-09-15T03:58:48.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.737+0000 7f6193954640 1 -- start start 2024-09-15T03:58:48.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.737+0000 7f6193954640 10 monclient: init 2024-09-15T03:58:48.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:48.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:58:48.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06e430 0x7f618c13e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 10 monclient(hunting): picked mon.noname-c con 0x7f618c06e430 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:48.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f618c06e800 0x7f618c12e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 10 monclient(hunting): picked mon.noname-b con 0x7f618c06e800 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:48.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f618c06ec20 0x7f618c117f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 10 monclient(hunting): picked mon.noname-a con 0x7f618c06ec20 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:48.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618c11ba70 con 0x7f618c06ec20 2024-09-15T03:58:48.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618c119f00 con 0x7f618c06e800 2024-09-15T03:58:48.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:48.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f618c13ee60 con 0x7f618c06e430 2024-09-15T03:58:48.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:48.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6193954640 10 monclient(hunting): authenticate will time out at 752.382186s 2024-09-15T03:58:48.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06e430 0x7f618c13e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06e430 0x7f618c13e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55210/0 (socket says 172.21.15.110:55210) 2024-09-15T03:58:48.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 1 -- 172.21.15.110:0/2854378475 learned_addr learned my addr 172.21.15.110:0/2854378475 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:48.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 10 monclient(hunting): get_auth_request con 0x7f618c06e430 auth_method 0 2024-09-15T03:58:48.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6191ef1640 1 --2- 172.21.15.110:0/2854378475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f618c06ec20 0x7f618c117f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6191ef1640 10 monclient(hunting): get_auth_request con 0x7f618c06ec20 auth_method 0 2024-09-15T03:58:48.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6191ef1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6191ef1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6191ef1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6190eef640 1 --2- 172.21.15.110:0/2854378475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f618c06e800 0x7f618c12e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6191ef1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:48.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6191ef1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:48.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6191ef1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:48.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6190eef640 10 monclient(hunting): get_auth_request con 0x7f618c06e800 auth_method 0 2024-09-15T03:58:48.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6190eef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:48.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6190eef640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:48.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6190eef640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:48.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2024-09-15T03:58:48.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 1 -- 172.21.15.110:0/2854378475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f618c06e800 msgr2=0x7f618c12e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 1 --2- 172.21.15.110:0/2854378475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f618c06e800 0x7f618c12e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 1 -- 172.21.15.110:0/2854378475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f618c06ec20 msgr2=0x7f618c117f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 1 --2- 172.21.15.110:0/2854378475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f618c06ec20 0x7f618c117f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 10 monclient: _finish_hunting 0 2024-09-15T03:58:48.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 1 monclient: found mon.noname-c 2024-09-15T03:58:48.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:48.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f61916f0640 1 -- 172.21.15.110:0/2854378475 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6184009ae0 con 0x7f618c06e430 2024-09-15T03:58:48.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.738+0000 7f6190eef640 1 --2- 172.21.15.110:0/2854378475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f618c06e800 0x7f618c12e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:58:48.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.740+0000 7f61916f0640 1 -- 172.21.15.110:0/2854378475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06e430 msgr2=0x7f618c13e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:48.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.740+0000 7f61916f0640 1 --2- 172.21.15.110:0/2854378475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06e430 0x7f618c13e920 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f618800ea20 tx=0x7f61880090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:48.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.740+0000 7f61817fa640 1 -- 172.21.15.110:0/2854378475 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188009600 con 0x7f618c06e430 2024-09-15T03:58:48.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.740+0000 7f61817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.740+0000 7f61817fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:48.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.740+0000 7f61817fa640 10 monclient: dump: 2024-09-15T03:58:48.755 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.755 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.755 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.755 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.755 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.755 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.756 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.756 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.756 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.756 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.740+0000 7f61817fa640 10 monclient: _finish_auth 0 2024-09-15T03:58:48.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.740+0000 7f61817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:48.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.740+0000 7f61817fa640 1 -- 172.21.15.110:0/2854378475 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618800cc20 con 0x7f618c06e430 2024-09-15T03:58:48.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.740+0000 7f6193954640 5 monclient: authenticate success, global_id 4322 2024-09-15T03:58:48.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.740+0000 7f6193954640 10 monclient: _renew_subs 2024-09-15T03:58:48.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.740+0000 7f6193954640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.740+0000 7f6193954640 1 -- 172.21.15.110:0/2854378475 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f618c09ae80 con 0x7f618c06e430 2024-09-15T03:58:48.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.741+0000 7f61817fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:48.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.741+0000 7f61817fa640 1 -- 172.21.15.110:0/2854378475 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188013470 con 0x7f618c06e430 2024-09-15T03:58:48.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.741+0000 7f61817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:48.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.741+0000 7f61817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:48.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.741+0000 7f61817fa640 10 monclient: dump: 2024-09-15T03:58:48.757 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:48.757 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:48.757 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.757 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:48.757 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:48.757 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:48.757 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:48.757 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:48.757 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:48.757 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:48.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.741+0000 7f6193954640 10 monclient: _renew_subs 2024-09-15T03:58:48.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.741+0000 7f6193954640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.741+0000 7f6193954640 1 -- 172.21.15.110:0/2854378475 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f618c119a60 con 0x7f618c06e430 2024-09-15T03:58:48.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.741+0000 7f61837fe640 4 set_mon_vals no callback set 2024-09-15T03:58:48.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.741+0000 7f61817fa640 1 -- 172.21.15.110:0/2854378475 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f6188013860 con 0x7f618c06e430 2024-09-15T03:58:48.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.741+0000 7f6193954640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:48.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.741+0000 7f6193954640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:48.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.741+0000 7f6193954640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:48.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.741+0000 7f6193954640 1 -- 172.21.15.110:0/2854378475 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618c13f790 con 0x7f618c06e430 2024-09-15T03:58:48.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.742+0000 7f61817fa640 1 --2- 172.21.15.110:0/2854378475 >> v2:172.21.15.177:6812/28201 conn(0x7f6170038750 0x7f617003ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:48.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.742+0000 7f61817fa640 1 -- 172.21.15.110:0/2854378475 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f618804b920 con 0x7f618c06e430 2024-09-15T03:58:48.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.742+0000 7f6190eef640 1 --2- 172.21.15.110:0/2854378475 >> v2:172.21.15.177:6812/28201 conn(0x7f6170038750 0x7f617003ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:48.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.742+0000 7f6190eef640 10 monclient: get_auth_request con 0x7f6170038750 auth_method 0 2024-09-15T03:58:48.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.744+0000 7f6190eef640 1 --2- 172.21.15.110:0/2854378475 >> v2:172.21.15.177:6812/28201 conn(0x7f6170038750 0x7f617003ac00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f617c002860 tx=0x7f617c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:48.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.746+0000 7f61817fa640 1 -- 172.21.15.110:0/2854378475 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6188023ad0 con 0x7f618c06e430 2024-09-15T03:58:48.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.746+0000 7f61817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:48.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.746+0000 7f61817fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:48.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.897+0000 7f6193954640 1 -- 172.21.15.110:0/2854378475 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f618c070950 con 0x7f6170038750 2024-09-15T03:58:48.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.899+0000 7f61817fa640 1 -- 172.21.15.110:0/2854378475 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+39252 (secure 0 0 0) 0x7f618c070950 con 0x7f6170038750 2024-09-15T03:58:48.900 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:48.901 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-09-15T03:58:48.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.901+0000 7f6193954640 1 -- 172.21.15.110:0/2854378475 >> v2:172.21.15.177:6812/28201 conn(0x7f6170038750 msgr2=0x7f617003ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.901+0000 7f6193954640 1 --2- 172.21.15.110:0/2854378475 >> v2:172.21.15.177:6812/28201 conn(0x7f6170038750 0x7f617003ac00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f617c002860 tx=0x7f617c009040 comp rx=0 tx=0).stop 2024-09-15T03:58:48.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.901+0000 7f6193954640 10 monclient: shutdown 2024-09-15T03:58:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.901+0000 7f6193954640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.901+0000 7f6193954640 1 -- 172.21.15.110:0/2854378475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06e430 msgr2=0x7f618c13e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.901+0000 7f6193954640 1 --2- 172.21.15.110:0/2854378475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06e430 0x7f618c13e920 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f618800ea20 tx=0x7f61880090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.901+0000 7f6193954640 1 -- 172.21.15.110:0/2854378475 shutdown_connections 2024-09-15T03:58:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.901+0000 7f6193954640 1 --2- 172.21.15.110:0/2854378475 >> v2:172.21.15.177:6812/28201 conn(0x7f6170038750 0x7f617003ac00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.901+0000 7f6193954640 1 --2- 172.21.15.110:0/2854378475 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f618c06ec20 0x7f618c117f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.901+0000 7f6193954640 1 --2- 172.21.15.110:0/2854378475 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f618c06e800 0x7f618c12e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.901+0000 7f6193954640 1 --2- 172.21.15.110:0/2854378475 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f618c06e430 0x7f618c13e920 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.901+0000 7f6193954640 1 -- 172.21.15.110:0/2854378475 >> 172.21.15.110:0/2854378475 conn(0x7f618c09d8f0 msgr2=0x7f618c131430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.901+0000 7f6193954640 1 -- 172.21.15.110:0/2854378475 shutdown_connections 2024-09-15T03:58:48.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:48.901+0000 7f6193954640 1 -- 172.21.15.110:0/2854378475 wait complete. 2024-09-15T03:58:48.913 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-15T03:58:48.363161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562077696,"kb_used":4777392,"kb_used_data":557300568,"kb_used_omap":443,"kb_used_meta":0,"kb_avail":557300304,"statfs":{"total":575567560704,"available":570675511296,"internally_reserved":270336,"allocated":570675781632,"data_stored":570675781632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":453772,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001072"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.072207+0000","last_change":"2024-09-15T03:58:44.072431+0000","last_active":"2024-09-15T03:58:44.072207+0000","last_peered":"2024-09-15T03:58:44.072207+0000","last_clean":"2024-09-15T03:58:44.072207+0000","last_became_active":"2024-09-15T03:58:42.157123+0000","last_became_peered":"2024-09-15T03:58:42.157123+0000","last_unstale":"2024-09-15T03:58:44.072207+0000","last_undegraded":"2024-09-15T03:58:44.072207+0000","last_fullsized":"2024-09-15T03:58:44.072207+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:46:37.797943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066518800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071760+0000","last_change":"2024-09-15T03:58:44.072012+0000","last_active":"2024-09-15T03:58:44.071760+0000","last_peered":"2024-09-15T03:58:44.071760+0000","last_clean":"2024-09-15T03:58:44.071760+0000","last_became_active":"2024-09-15T03:58:42.069324+0000","last_became_peered":"2024-09-15T03:58:42.069324+0000","last_unstale":"2024-09-15T03:58:44.071760+0000","last_undegraded":"2024-09-15T03:58:44.071760+0000","last_fullsized":"2024-09-15T03:58:44.071760+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:44:04.033148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040535300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071964+0000","last_change":"2024-09-15T03:58:44.072166+0000","last_active":"2024-09-15T03:58:44.071964+0000","last_peered":"2024-09-15T03:58:44.071964+0000","last_clean":"2024-09-15T03:58:44.071964+0000","last_became_active":"2024-09-15T03:58:42.155521+0000","last_became_peered":"2024-09-15T03:58:42.155521+0000","last_unstale":"2024-09-15T03:58:44.071964+0000","last_undegraded":"2024-09-15T03:58:44.071964+0000","last_fullsized":"2024-09-15T03:58:44.071964+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:42:01.920640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040527199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071921+0000","last_change":"2024-09-15T03:58:44.072137+0000","last_active":"2024-09-15T03:58:44.071921+0000","last_peered":"2024-09-15T03:58:44.071921+0000","last_clean":"2024-09-15T03:58:44.071921+0000","last_became_active":"2024-09-15T03:58:42.155317+0000","last_became_peered":"2024-09-15T03:58:42.155317+0000","last_unstale":"2024-09-15T03:58:44.071921+0000","last_undegraded":"2024-09-15T03:58:44.071921+0000","last_fullsized":"2024-09-15T03:58:44.071921+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:55:28.163584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041321199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.074749+0000","last_change":"2024-09-15T03:58:44.074749+0000","last_active":"2024-09-15T03:58:44.074749+0000","last_peered":"2024-09-15T03:58:44.074749+0000","last_clean":"2024-09-15T03:58:44.074749+0000","last_became_active":"2024-09-15T03:58:42.071376+0000","last_became_peered":"2024-09-15T03:58:42.071376+0000","last_unstale":"2024-09-15T03:58:44.074749+0000","last_undegraded":"2024-09-15T03:58:44.074749+0000","last_fullsized":"2024-09-15T03:58:44.074749+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:42:52.640034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080084100000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071218+0000","last_change":"2024-09-15T03:58:44.071453+0000","last_active":"2024-09-15T03:58:44.071218+0000","last_peered":"2024-09-15T03:58:44.071218+0000","last_clean":"2024-09-15T03:58:44.071218+0000","last_became_active":"2024-09-15T03:58:42.067012+0000","last_became_peered":"2024-09-15T03:58:42.067012+0000","last_unstale":"2024-09-15T03:58:44.071218+0000","last_undegraded":"2024-09-15T03:58:44.071218+0000","last_fullsized":"2024-09-15T03:58:44.071218+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:04:08.843151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040766700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071926+0000","last_change":"2024-09-15T03:58:44.072155+0000","last_active":"2024-09-15T03:58:44.071926+0000","last_peered":"2024-09-15T03:58:44.071926+0000","last_clean":"2024-09-15T03:58:44.071926+0000","last_became_active":"2024-09-15T03:58:42.069300+0000","last_became_peered":"2024-09-15T03:58:42.069300+0000","last_unstale":"2024-09-15T03:58:44.071926+0000","last_undegraded":"2024-09-15T03:58:44.071926+0000","last_fullsized":"2024-09-15T03:58:44.071926+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:28:34.591535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036612400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.074041+0000","last_change":"2024-09-15T03:58:44.074233+0000","last_active":"2024-09-15T03:58:44.074041+0000","last_peered":"2024-09-15T03:58:44.074041+0000","last_clean":"2024-09-15T03:58:44.074041+0000","last_became_active":"2024-09-15T03:58:42.071652+0000","last_became_peered":"2024-09-15T03:58:42.071652+0000","last_unstale":"2024-09-15T03:58:44.074041+0000","last_undegraded":"2024-09-15T03:58:44.074041+0000","last_fullsized":"2024-09-15T03:58:44.074041+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:31:44.288660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030212199999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":103,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T03:58:43.157400+0000","last_change":"2024-09-15T03:58:38.057397+0000","last_active":"2024-09-15T03:58:43.157400+0000","last_peered":"2024-09-15T03:58:43.157400+0000","last_clean":"2024-09-15T03:58:43.157400+0000","last_became_active":"2024-09-15T03:58:38.056673+0000","last_became_peered":"2024-09-15T03:58:38.056673+0000","last_unstale":"2024-09-15T03:58:43.157400+0000","last_undegraded":"2024-09-15T03:58:43.157400+0000","last_fullsized":"2024-09-15T03:58:43.157400+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_clean_scrub_stamp":"2024-09-15T03:58:37.036429+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:38:58.792683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795816,"kb_used_data":92883832,"kb_used_omap":56,"kb_used_meta":0,"kb_avail":92883800,"statfs":{"total":95927926784,"available":95113011200,"internally_reserved":32768,"allocated":95113043968,"data_stored":95113043968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58360,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795844,"kb_used_data":92883816,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92883772,"statfs":{"total":95927926784,"available":95112982528,"internally_reserved":45056,"allocated":95113027584,"data_stored":95113027584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48544,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797028,"kb_used_data":92882636,"kb_used_omap":109,"kb_used_meta":0,"kb_avail":92882588,"statfs":{"total":95927926784,"available":95111770112,"internally_reserved":49152,"allocated":95111819264,"data_stored":95111819264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112614,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795784,"kb_used_data":92883856,"kb_used_omap":55,"kb_used_meta":0,"kb_avail":92883832,"statfs":{"total":95927926784,"available":95113043968,"internally_reserved":24576,"allocated":95113068544,"data_stored":95113068544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":56457,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795840,"kb_used_data":92883828,"kb_used_omap":74,"kb_used_meta":0,"kb_avail":92883776,"statfs":{"total":95927926784,"available":95112986624,"internally_reserved":53248,"allocated":95113039872,"data_stored":95113039872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76777,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797080,"kb_used_data":92882600,"kb_used_omap":98,"kb_used_meta":0,"kb_avail":92882536,"statfs":{"total":95927926784,"available":95111716864,"internally_reserved":65536,"allocated":95111782400,"data_stored":95111782400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101020,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]}]}],"pool_statfs":[]}} 2024-09-15T03:58:48.913 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T03:58:48.914 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T03:58:48.914 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-15T03:58:48.914 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-15T03:58:49.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.004+0000 7f589159d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:49.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.004+0000 7f589159d640 10 monclient: build_initial_monmap 2024-09-15T03:58:49.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.004+0000 7f589159d640 10 monclient: monmap: 2024-09-15T03:58:49.005 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:49.006 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.006 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:49.005704+0000 2024-09-15T03:58:49.006 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:49.005704+0000 2024-09-15T03:58:49.006 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:49.006 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.006 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:49.006 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:49.006 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:49.006 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:49.006 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:49.006 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 1 Processor -- start 2024-09-15T03:58:49.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 1 -- start start 2024-09-15T03:58:49.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 10 monclient: init 2024-09-15T03:58:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T03:58:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c0a1bf0 0x7f588c131810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 10 monclient(hunting): picked mon.noname-b con 0x7f588c0a1bf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f588c09e780 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f588c136a40 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588c09c9b0 con 0x7f588c136a40 2024-09-15T03:58:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588c0a1fc0 con 0x7f588c09e780 2024-09-15T03:58:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588c0a0c20 con 0x7f588c0a1bf0 2024-09-15T03:58:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.005+0000 7f589159d640 10 monclient(hunting): authenticate will time out at 752.649958s 2024-09-15T03:58:49.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c0a1bf0 0x7f588c131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588b7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f588c136a40 0x7f588c138e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:33892/0 (socket says 172.21.15.110:33892) 2024-09-15T03:58:49.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588b7fe640 1 -- 172.21.15.110:0/626948126 learned_addr learned my addr 172.21.15.110:0/626948126 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:49.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 10 monclient(hunting): get_auth_request con 0x7f588c0a1bf0 auth_method 0 2024-09-15T03:58:49.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:49.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:49.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:49.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-09-15T03:58:49.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 1 -- 172.21.15.110:0/626948126 >> v1:172.21.15.177:6789/0 conn(0x7f588c09e780 legacy=0x7f588c09eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:49.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 1 -- 172.21.15.110:0/626948126 >> v1:172.21.15.110:6789/0 conn(0x7f588c136a40 legacy=0x7f588c138e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:49.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 10 monclient: _finish_hunting 0 2024-09-15T03:58:49.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 1 monclient: found mon.noname-b 2024-09-15T03:58:49.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:49.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.006+0000 7f588affd640 1 -- 172.21.15.110:0/626948126 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588c0a0900 con 0x7f588c0a1bf0 2024-09-15T03:58:49.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f588affd640 1 -- 172.21.15.110:0/626948126 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c0a1bf0 msgr2=0x7f588c131810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:49.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f588affd640 1 --2- 172.21.15.110:0/626948126 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c0a1bf0 0x7f588c131810 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f588c0682d0 tx=0x7f587c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df5692a05c11f979 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:49.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f588a7fc640 1 -- 172.21.15.110:0/626948126 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c032440 con 0x7f588c0a1bf0 2024-09-15T03:58:49.016 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f588a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f588a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f588a7fc640 10 monclient: dump: 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f588a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f588a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:49.017 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f588a7fc640 1 -- 172.21.15.110:0/626948126 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587c0325a0 con 0x7f588c0a1bf0 2024-09-15T03:58:49.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f589159d640 5 monclient: authenticate success, global_id 4325 2024-09-15T03:58:49.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f589159d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:49.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f588a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:49.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f589159d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:49.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f589159d640 4 set_mon_vals no callback set 2024-09-15T03:58:49.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f588a7fc640 1 -- 172.21.15.110:0/626948126 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c032940 con 0x7f588c0a1bf0 2024-09-15T03:58:49.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f588a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f588a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:49.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f588a7fc640 10 monclient: dump: 2024-09-15T03:58:49.018 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.018 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.018 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.018 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.019 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.019 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.019 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.019 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.019 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.019 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f589159d640 10 monclient: shutdown 2024-09-15T03:58:49.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f589159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:49.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f589159d640 1 -- 172.21.15.110:0/626948126 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c0a1bf0 msgr2=0x7f588c131810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f589159d640 1 --2- 172.21.15.110:0/626948126 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c0a1bf0 0x7f588c131810 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f588c0682d0 tx=0x7f587c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f589159d640 1 -- 172.21.15.110:0/626948126 shutdown_connections 2024-09-15T03:58:49.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f589159d640 1 --2- 172.21.15.110:0/626948126 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c0a1bf0 0x7f588c131810 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f589159d640 1 -- 172.21.15.110:0/626948126 >> 172.21.15.110:0/626948126 conn(0x7f588c062dd0 msgr2=0x7f588c0631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f589159d640 1 -- 172.21.15.110:0/626948126 shutdown_connections 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.009+0000 7f589159d640 1 -- 172.21.15.110:0/626948126 wait complete. 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.010+0000 7f589159d640 10 monclient: build_initial_monmap 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.010+0000 7f589159d640 10 monclient: monmap: 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.010+0000 7f589159d640 1 Processor -- start 2024-09-15T03:58:49.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.010+0000 7f589159d640 1 -- start start 2024-09-15T03:58:49.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.010+0000 7f589159d640 10 monclient: init 2024-09-15T03:58:49.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:49.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:58:49.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f588c09e780 0x7f588c09a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 10 monclient(hunting): picked mon.noname-b con 0x7f588c09e780 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:49.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f588c0a1bf0 0x7f588c096680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 10 monclient(hunting): picked mon.noname-a con 0x7f588c0a1bf0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:49.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c136a40 0x7f588c096bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 10 monclient(hunting): picked mon.noname-c con 0x7f588c136a40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:49.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c098280 con 0x7f588c0a1bf0 2024-09-15T03:58:49.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588c09c280 con 0x7f588c09e780 2024-09-15T03:58:49.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588c099a40 con 0x7f588c136a40 2024-09-15T03:58:49.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:49.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f589159d640 10 monclient(hunting): authenticate will time out at 752.655110s 2024-09-15T03:58:49.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f5883fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f588c0a1bf0 0x7f588c096680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c136a40 0x7f588c096bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c136a40 0x7f588c096bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:55226/0 (socket says 172.21.15.110:55226) 2024-09-15T03:58:49.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 1 -- 172.21.15.110:0/4273915446 learned_addr learned my addr 172.21.15.110:0/4273915446 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:49.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f5883fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f588c0a1bf0 0x7f588c096680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56692/0 (socket says 172.21.15.110:56692) 2024-09-15T03:58:49.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 10 monclient(hunting): get_auth_request con 0x7f588c136a40 auth_method 0 2024-09-15T03:58:49.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f5883fff640 10 monclient(hunting): get_auth_request con 0x7f588c0a1bf0 auth_method 0 2024-09-15T03:58:49.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f5883fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f5883fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f5883fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:49.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:49.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:49.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f5883fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:49.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f5883fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:49.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f5883fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:49.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588affd640 1 --2- 172.21.15.110:0/4273915446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f588c09e780 0x7f588c09a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-09-15T03:58:49.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 1 -- 172.21.15.110:0/4273915446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f588c09e780 msgr2=0x7f588c09a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.023 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 1 --2- 172.21.15.110:0/4273915446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f588c09e780 0x7f588c09a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 1 -- 172.21.15.110:0/4273915446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f588c0a1bf0 msgr2=0x7f588c096680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 1 --2- 172.21.15.110:0/4273915446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f588c0a1bf0 0x7f588c096680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 10 monclient: _finish_hunting 0 2024-09-15T03:58:49.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 1 monclient: found mon.noname-c 2024-09-15T03:58:49.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:49.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f588b7fe640 1 -- 172.21.15.110:0/4273915446 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587c009390 con 0x7f588c136a40 2024-09-15T03:58:49.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.011+0000 7f5883fff640 1 --2- 172.21.15.110:0/4273915446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f588c0a1bf0 0x7f588c096680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:49.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f588b7fe640 1 -- 172.21.15.110:0/4273915446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c136a40 msgr2=0x7f588c096bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:49.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f588b7fe640 1 --2- 172.21.15.110:0/4273915446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c136a40 0x7f588c096bc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f588400ccc0 tx=0x7f5884005370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:49.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f5882ffd640 1 -- 172.21.15.110:0/4273915446 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588400ce80 con 0x7f588c136a40 2024-09-15T03:58:49.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f5882ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.024 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f5882ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f5882ffd640 10 monclient: dump: 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f5882ffd640 10 monclient: _finish_auth 0 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f5882ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f5882ffd640 1 -- 172.21.15.110:0/4273915446 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f588400aa60 con 0x7f588c136a40 2024-09-15T03:58:49.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f589159d640 5 monclient: authenticate success, global_id 4328 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f589159d640 10 monclient: _renew_subs 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f589159d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f589159d640 1 -- 172.21.15.110:0/4273915446 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588c099fd0 con 0x7f588c136a40 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f5882ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f5882ffd640 1 -- 172.21.15.110:0/4273915446 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588400ae20 con 0x7f588c136a40 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f5882ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f5882ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f5882ffd640 10 monclient: dump: 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.026 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f5889ffb640 4 set_mon_vals no callback set 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.013+0000 7f589159d640 10 monclient: _renew_subs 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.014+0000 7f589159d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.014+0000 7f589159d640 1 -- 172.21.15.110:0/4273915446 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588c098d90 con 0x7f588c136a40 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.014+0000 7f5882ffd640 1 -- 172.21.15.110:0/4273915446 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f5884007300 con 0x7f588c136a40 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.014+0000 7f589159d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.014+0000 7f589159d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.014+0000 7f589159d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.014+0000 7f589159d640 1 -- 172.21.15.110:0/4273915446 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588c070950 con 0x7f588c136a40 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.014+0000 7f5882ffd640 1 --2- 172.21.15.110:0/4273915446 >> v2:172.21.15.177:6812/28201 conn(0x7f586c038700 0x7f586c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.015+0000 7f5882ffd640 1 -- 172.21.15.110:0/4273915446 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f588404b710 con 0x7f588c136a40 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.015+0000 7f588affd640 1 --2- 172.21.15.110:0/4273915446 >> v2:172.21.15.177:6812/28201 conn(0x7f586c038700 0x7f586c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.015+0000 7f588affd640 10 monclient: get_auth_request con 0x7f586c038700 auth_method 0 2024-09-15T03:58:49.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.017+0000 7f588affd640 1 --2- 172.21.15.110:0/4273915446 >> v2:172.21.15.177:6812/28201 conn(0x7f586c038700 0x7f586c03abb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f588c09d670 tx=0x7f587c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:49.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.019+0000 7f5882ffd640 1 -- 172.21.15.110:0/4273915446 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f58840075b0 con 0x7f588c136a40 2024-09-15T03:58:49.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.019+0000 7f5882ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:49.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.019+0000 7f5882ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:49.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.208+0000 7f589159d640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-15T03:58:49.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.208+0000 7f589159d640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-15T03:58:49.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.208+0000 7f589159d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.208+0000 7f589159d640 1 -- 172.21.15.110:0/4273915446 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f588c0a26c0 con 0x7f588c136a40 2024-09-15T03:58:49.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.209+0000 7f5882ffd640 1 -- 172.21.15.110:0/4273915446 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f5884010af0 con 0x7f588c136a40 2024-09-15T03:58:49.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.209+0000 7f5882ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-15T03:58:49.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.209+0000 7f5882ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:49.210 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:49.211 INFO:teuthology.orchestra.run.smithi110.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T03:58:49.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.211+0000 7f589159d640 1 -- 172.21.15.110:0/4273915446 >> v2:172.21.15.177:6812/28201 conn(0x7f586c038700 msgr2=0x7f586c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.211+0000 7f589159d640 1 --2- 172.21.15.110:0/4273915446 >> v2:172.21.15.177:6812/28201 conn(0x7f586c038700 0x7f586c03abb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f588c09d670 tx=0x7f587c032000 comp rx=0 tx=0).stop 2024-09-15T03:58:49.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.211+0000 7f589159d640 10 monclient: shutdown 2024-09-15T03:58:49.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.211+0000 7f589159d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:49.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.211+0000 7f589159d640 1 -- 172.21.15.110:0/4273915446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c136a40 msgr2=0x7f588c096bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.211+0000 7f589159d640 1 --2- 172.21.15.110:0/4273915446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c136a40 0x7f588c096bc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f588400ccc0 tx=0x7f5884005370 comp rx=0 tx=0).stop 2024-09-15T03:58:49.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.211+0000 7f589159d640 1 -- 172.21.15.110:0/4273915446 shutdown_connections 2024-09-15T03:58:49.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.211+0000 7f589159d640 1 --2- 172.21.15.110:0/4273915446 >> v2:172.21.15.177:6812/28201 conn(0x7f586c038700 0x7f586c03abb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.211+0000 7f589159d640 1 --2- 172.21.15.110:0/4273915446 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f588c136a40 0x7f588c096bc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.211+0000 7f589159d640 1 --2- 172.21.15.110:0/4273915446 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f588c0a1bf0 0x7f588c096680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.211+0000 7f589159d640 1 --2- 172.21.15.110:0/4273915446 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f588c09e780 0x7f588c09a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.211+0000 7f589159d640 1 -- 172.21.15.110:0/4273915446 >> 172.21.15.110:0/4273915446 conn(0x7f588c062dd0 msgr2=0x7f588c0a4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:49.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.211+0000 7f589159d640 1 -- 172.21.15.110:0/4273915446 shutdown_connections 2024-09-15T03:58:49.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.211+0000 7f589159d640 1 -- 172.21.15.110:0/4273915446 wait complete. 2024-09-15T03:58:49.222 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-15T03:58:49.222 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-15T03:58:49.242 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-15T03:58:49.243 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:58:49.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.334+0000 7f9e90094640 10 monclient: get_monmap_and_config 2024-09-15T03:58:49.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.334+0000 7f9e90094640 10 monclient: build_initial_monmap 2024-09-15T03:58:49.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.334+0000 7f9e90094640 10 monclient: monmap: 2024-09-15T03:58:49.335 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:49.335 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.335 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:49.335505+0000 2024-09-15T03:58:49.336 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:49.335505+0000 2024-09-15T03:58:49.336 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:49.336 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.336 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:49.336 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:49.336 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:49.336 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:49.336 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:49.336 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 1 Processor -- start 2024-09-15T03:58:49.338 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 1 -- start start 2024-09-15T03:58:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 10 monclient: init 2024-09-15T03:58:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-15T03:58:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8806e590 0x7f9e8809ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 10 monclient(hunting): picked mon.noname-b con 0x7f9e8806e590 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e8809e2f0 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9e8809e6c0 0x7f9e8812fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 10 monclient(hunting): picked mon.noname-a con 0x7f9e8809e6c0 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e8809f340 con 0x7f9e8809e2f0 2024-09-15T03:58:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8806dab0 con 0x7f9e8809e6c0 2024-09-15T03:58:49.339 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8806e960 con 0x7f9e8806e590 2024-09-15T03:58:49.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:49.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e90094640 10 monclient(hunting): authenticate will time out at 752.979812s 2024-09-15T03:58:49.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e8de30640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8806e590 0x7f9e8809ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e8d62f640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9e8809e2f0 0x7f9e8812f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:34132/0 (socket says 172.21.15.110:34132) 2024-09-15T03:58:49.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e8d62f640 1 -- 172.21.15.110:0/3657579327 learned_addr learned my addr 172.21.15.110:0/3657579327 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:49.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e8de30640 10 monclient(hunting): get_auth_request con 0x7f9e8806e590 auth_method 0 2024-09-15T03:58:49.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e8e631640 1 --2- 172.21.15.110:0/3657579327 >> v2:172.21.15.110:3300/0 conn(0x7f9e8809e6c0 0x7f9e8812fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e8de30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e8de30640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e8de30640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e8e631640 10 monclient(hunting): get_auth_request con 0x7f9e8809e6c0 auth_method 0 2024-09-15T03:58:49.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e8e631640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.340 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e8e631640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.335+0000 7f9e8e631640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8de30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:49.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8de30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:49.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8de30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:49.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8e631640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:49.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8e631640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:49.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8e631640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:49.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8de30640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2024-09-15T03:58:49.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8de30640 1 -- 172.21.15.110:0/3657579327 >> v2:172.21.15.110:3300/0 conn(0x7f9e8809e6c0 msgr2=0x7f9e8812fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:49.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8de30640 1 --2- 172.21.15.110:0/3657579327 >> v2:172.21.15.110:3300/0 conn(0x7f9e8809e6c0 0x7f9e8812fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8de30640 1 -- 172.21.15.110:0/3657579327 >> v1:172.21.15.110:6789/0 conn(0x7f9e8809e2f0 legacy=0x7f9e8812f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:49.341 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8de30640 10 monclient: _finish_hunting 0 2024-09-15T03:58:49.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8de30640 1 monclient: found mon.noname-b 2024-09-15T03:58:49.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8de30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:49.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8de30640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.342 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.336+0000 7f9e8de30640 1 -- 172.21.15.110:0/3657579327 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e8806f040 con 0x7f9e8806e590 2024-09-15T03:58:49.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.338+0000 7f9e8de30640 1 -- 172.21.15.110:0/3657579327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8806e590 msgr2=0x7f9e8809ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:49.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.338+0000 7f9e8de30640 1 --2- 172.21.15.110:0/3657579327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8806e590 0x7f9e8809ddb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9e84009880 tx=0x7f9e840090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a944d1cb2542407f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:49.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e8ce2e640 1 -- 172.21.15.110:0/3657579327 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8402ae20 con 0x7f9e8806e590 2024-09-15T03:58:49.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e8ce2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e8ce2e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:49.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e8ce2e640 10 monclient: dump: 2024-09-15T03:58:49.349 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.349 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.349 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.349 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.349 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.349 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.349 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.349 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.349 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.349 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e8ce2e640 10 monclient: _finish_auth 0 2024-09-15T03:58:49.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e8ce2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:49.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e90094640 5 monclient: authenticate success, global_id 4331 2024-09-15T03:58:49.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e8ce2e640 1 -- 172.21.15.110:0/3657579327 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e84032440 con 0x7f9e8806e590 2024-09-15T03:58:49.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e90094640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:49.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e8ce2e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:49.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e90094640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:49.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e90094640 4 set_mon_vals no callback set 2024-09-15T03:58:49.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e8ce2e640 1 -- 172.21.15.110:0/3657579327 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840327e0 con 0x7f9e8806e590 2024-09-15T03:58:49.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e8ce2e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e8ce2e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:49.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e8ce2e640 10 monclient: dump: 2024-09-15T03:58:49.350 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.350 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.350 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.350 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.350 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.350 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.351 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.351 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.351 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.351 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e90094640 10 monclient: shutdown 2024-09-15T03:58:49.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e90094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:49.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e90094640 1 -- 172.21.15.110:0/3657579327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8806e590 msgr2=0x7f9e8809ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e90094640 1 --2- 172.21.15.110:0/3657579327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8806e590 0x7f9e8809ddb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9e84009880 tx=0x7f9e840090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e90094640 1 -- 172.21.15.110:0/3657579327 shutdown_connections 2024-09-15T03:58:49.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e90094640 1 --2- 172.21.15.110:0/3657579327 >> v2:172.21.15.110:3300/0 conn(0x7f9e8809e6c0 0x7f9e8812fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e90094640 1 --2- 172.21.15.110:0/3657579327 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8806e590 0x7f9e8809ddb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e90094640 1 -- 172.21.15.110:0/3657579327 >> 172.21.15.110:0/3657579327 conn(0x7f9e880639f0 msgr2=0x7f9e880a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:49.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e90094640 1 -- 172.21.15.110:0/3657579327 shutdown_connections 2024-09-15T03:58:49.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.339+0000 7f9e90094640 1 -- 172.21.15.110:0/3657579327 wait complete. 2024-09-15T03:58:49.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 10 monclient: build_initial_monmap 2024-09-15T03:58:49.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 10 monclient: monmap: 2024-09-15T03:58:49.352 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:49.352 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:49.352 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:49.352 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:49.352 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:49.352 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.352 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:49.352 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:49.353 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:49.353 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 1 Processor -- start 2024-09-15T03:58:49.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 1 -- start start 2024-09-15T03:58:49.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 10 monclient: init 2024-09-15T03:58:49.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:49.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:58:49.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e8806e590 0x7f9e8813eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 10 monclient(hunting): picked mon.noname-b con 0x7f9e8806e590 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:49.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8809e2f0 0x7f9e8812f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 10 monclient(hunting): picked mon.noname-c con 0x7f9e8809e2f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:49.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e8809e6c0 0x7f9e88119200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 10 monclient(hunting): picked mon.noname-a con 0x7f9e8809e6c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:49.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8811ce70 con 0x7f9e8809e6c0 2024-09-15T03:58:49.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8809cda0 con 0x7f9e8806e590 2024-09-15T03:58:49.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8809c080 con 0x7f9e8809e2f0 2024-09-15T03:58:49.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:49.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.340+0000 7f9e90094640 10 monclient(hunting): authenticate will time out at 752.984988s 2024-09-15T03:58:49.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8809e2f0 0x7f9e8812f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8e631640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e8809e6c0 0x7f9e88119200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8809e2f0 0x7f9e8812f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43456/0 (socket says 172.21.15.110:43456) 2024-09-15T03:58:49.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8e631640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e8809e6c0 0x7f9e88119200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53086/0 (socket says 172.21.15.110:53086) 2024-09-15T03:58:49.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 1 -- 172.21.15.110:0/1213138913 learned_addr learned my addr 172.21.15.110:0/1213138913 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:49.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 10 monclient(hunting): get_auth_request con 0x7f9e8809e2f0 auth_method 0 2024-09-15T03:58:49.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8e631640 10 monclient(hunting): get_auth_request con 0x7f9e8809e6c0 auth_method 0 2024-09-15T03:58:49.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8e631640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8e631640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8e631640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:49.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:49.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:49.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8e631640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:49.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8e631640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:49.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8e631640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:49.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8de30640 1 --2- 172.21.15.110:0/1213138913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e8806e590 0x7f9e8813eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8de30640 10 monclient(hunting): get_auth_request con 0x7f9e8806e590 auth_method 0 2024-09-15T03:58:49.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8de30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8de30640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8de30640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2024-09-15T03:58:49.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 1 -- 172.21.15.110:0/1213138913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e8806e590 msgr2=0x7f9e8813eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 1 --2- 172.21.15.110:0/1213138913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e8806e590 0x7f9e8813eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 1 -- 172.21.15.110:0/1213138913 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e8809e6c0 msgr2=0x7f9e88119200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 1 --2- 172.21.15.110:0/1213138913 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e8809e6c0 0x7f9e88119200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 10 monclient: _finish_hunting 0 2024-09-15T03:58:49.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 1 monclient: found mon.noname-c 2024-09-15T03:58:49.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:49.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8d62f640 1 -- 172.21.15.110:0/1213138913 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e84009390 con 0x7f9e8809e2f0 2024-09-15T03:58:49.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.341+0000 7f9e8e631640 1 --2- 172.21.15.110:0/1213138913 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e8809e6c0 0x7f9e88119200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:49.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e8d62f640 1 -- 172.21.15.110:0/1213138913 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8809e2f0 msgr2=0x7f9e8812f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:49.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e8d62f640 1 --2- 172.21.15.110:0/1213138913 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8809e2f0 0x7f9e8812f700 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800c610 tx=0x7f9e78004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:49.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e7dffb640 1 -- 172.21.15.110:0/1213138913 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e78005cb0 con 0x7f9e8809e2f0 2024-09-15T03:58:49.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e7dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:49.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e7dffb640 10 monclient: dump: 2024-09-15T03:58:49.357 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.357 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.357 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.358 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.358 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.358 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.358 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.358 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.358 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.358 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e7dffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:49.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:49.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e90094640 5 monclient: authenticate success, global_id 4334 2024-09-15T03:58:49.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e90094640 10 monclient: _renew_subs 2024-09-15T03:58:49.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e7dffb640 1 -- 172.21.15.110:0/1213138913 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e78005e30 con 0x7f9e8809e2f0 2024-09-15T03:58:49.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e90094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e90094640 1 -- 172.21.15.110:0/1213138913 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8811b200 con 0x7f9e8809e2f0 2024-09-15T03:58:49.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e7dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:49.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e7dffb640 1 -- 172.21.15.110:0/1213138913 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e78007340 con 0x7f9e8809e2f0 2024-09-15T03:58:49.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e7dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:49.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e7dffb640 10 monclient: dump: 2024-09-15T03:58:49.359 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.359 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.359 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.359 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.359 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.359 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.359 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.360 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.360 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.360 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e7ffff640 4 set_mon_vals no callback set 2024-09-15T03:58:49.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.343+0000 7f9e90094640 10 monclient: _renew_subs 2024-09-15T03:58:49.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.344+0000 7f9e90094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.344+0000 7f9e90094640 1 -- 172.21.15.110:0/1213138913 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8811c820 con 0x7f9e8809e2f0 2024-09-15T03:58:49.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.344+0000 7f9e7dffb640 1 -- 172.21.15.110:0/1213138913 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f9e7801b400 con 0x7f9e8809e2f0 2024-09-15T03:58:49.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.344+0000 7f9e7dffb640 1 --2- 172.21.15.110:0/1213138913 >> v2:172.21.15.177:6812/28201 conn(0x7f9e6c038700 0x7f9e6c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.344+0000 7f9e7dffb640 1 -- 172.21.15.110:0/1213138913 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f9e7804b170 con 0x7f9e8809e2f0 2024-09-15T03:58:49.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.344+0000 7f9e90094640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:49.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.344+0000 7f9e90094640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:49.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.344+0000 7f9e90094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.344+0000 7f9e8de30640 1 --2- 172.21.15.110:0/1213138913 >> v2:172.21.15.177:6812/28201 conn(0x7f9e6c038700 0x7f9e6c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.344+0000 7f9e90094640 1 -- 172.21.15.110:0/1213138913 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e8811cb90 con 0x7f9e8809e2f0 2024-09-15T03:58:49.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.344+0000 7f9e8de30640 10 monclient: get_auth_request con 0x7f9e6c038700 auth_method 0 2024-09-15T03:58:49.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.347+0000 7f9e8de30640 1 --2- 172.21.15.110:0/1213138913 >> v2:172.21.15.177:6812/28201 conn(0x7f9e6c038700 0x7f9e6c03abb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9e84034000 tx=0x7f9e84032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:49.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.349+0000 7f9e7dffb640 1 -- 172.21.15.110:0/1213138913 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9e78012760 con 0x7f9e8809e2f0 2024-09-15T03:58:49.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.349+0000 7f9e7dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:49.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.349+0000 7f9e7dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:49.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.503+0000 7f9e90094640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:49.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.503+0000 7f9e90094640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:49.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.503+0000 7f9e90094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.503+0000 7f9e90094640 1 -- 172.21.15.110:0/1213138913 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e8811adb0 con 0x7f9e8809e2f0 2024-09-15T03:58:49.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.503+0000 7f9e7dffb640 1 -- 172.21.15.110:0/1213138913 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+10065 (secure 0 0 0) 0x7f9e8811adb0 con 0x7f9e8809e2f0 2024-09-15T03:58:49.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.503+0000 7f9e7dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:58:49.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.503+0000 7f9e7dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:49.505 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:49.505 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":20,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","created":"2024-09-15T03:58:28.265907+0000","modified":"2024-09-15T03:58:48.042558+0000","last_up_change":"2024-09-15T03:58:35.032799+0000","last_in_change":"2024-09-15T03:58:31.538597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:58:36.398831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:58:41.004777+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"105a3b3d-e410-47fb-ac54-6a5d5da4a46f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":28367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":28367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":28367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":28367}]},"public_addr":"172.21.15.110:6808/28367","cluster_addr":"172.21.15.110:6809/28367","heartbeat_back_addr":"172.21.15.110:6811/28367","heartbeat_front_addr":"172.21.15.110:6810/28367","state":["exists","up"]},{"osd":1,"uuid":"77f0f205-6c64-495a-8edd-043de180efa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":28369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":28369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":28369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":28369}]},"public_addr":"172.21.15.110:6800/28369","cluster_addr":"172.21.15.110:6801/28369","heartbeat_back_addr":"172.21.15.110:6803/28369","heartbeat_front_addr":"172.21.15.110:6802/28369","state":["exists","up"]},{"osd":2,"uuid":"e8a42031-303e-4ad0-9896-e5b84cc95cfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":28368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":28368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":28368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":28368}]},"public_addr":"172.21.15.110:6804/28368","cluster_addr":"172.21.15.110:6805/28368","heartbeat_back_addr":"172.21.15.110:6807/28368","heartbeat_front_addr":"172.21.15.110:6806/28368","state":["exists","up"]},{"osd":3,"uuid":"0d65db82-4a0e-4f31-a10a-dcd71dcb269f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":28628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6805","nonce":28628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6807","nonce":28628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":28628}]},"public_addr":"172.21.15.177:6804/28628","cluster_addr":"172.21.15.177:6805/28628","heartbeat_back_addr":"172.21.15.177:6807/28628","heartbeat_front_addr":"172.21.15.177:6806/28628","state":["exists","up"]},{"osd":4,"uuid":"e95e4f77-578d-4d03-bb4d-ee72b43bc227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":28629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6801","nonce":28629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6803","nonce":28629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":28629}]},"public_addr":"172.21.15.177:6800/28629","cluster_addr":"172.21.15.177:6801/28629","heartbeat_back_addr":"172.21.15.177:6803/28629","heartbeat_front_addr":"172.21.15.177:6802/28629","state":["exists","up"]},{"osd":5,"uuid":"f16ffee6-1e25-432c-b6c2-909a72ce24a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":28630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6809","nonce":28630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6811","nonce":28630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":28630}]},"public_addr":"172.21.15.177:6808/28630","cluster_addr":"172.21.15.177:6809/28630","heartbeat_back_addr":"172.21.15.177:6811/28630","heartbeat_front_addr":"172.21.15.177:6810/28630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.117560+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.137736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.130715+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.173663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.130009+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.172505+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:58:49.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.505+0000 7f9e90094640 1 -- 172.21.15.110:0/1213138913 >> v2:172.21.15.177:6812/28201 conn(0x7f9e6c038700 msgr2=0x7f9e6c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.505+0000 7f9e90094640 1 --2- 172.21.15.110:0/1213138913 >> v2:172.21.15.177:6812/28201 conn(0x7f9e6c038700 0x7f9e6c03abb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9e84034000 tx=0x7f9e84032000 comp rx=0 tx=0).stop 2024-09-15T03:58:49.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.505+0000 7f9e90094640 10 monclient: shutdown 2024-09-15T03:58:49.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.505+0000 7f9e90094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:49.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.505+0000 7f9e90094640 1 -- 172.21.15.110:0/1213138913 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8809e2f0 msgr2=0x7f9e8812f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.505+0000 7f9e90094640 1 --2- 172.21.15.110:0/1213138913 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8809e2f0 0x7f9e8812f700 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9e7800c610 tx=0x7f9e78004960 comp rx=0 tx=0).stop 2024-09-15T03:58:49.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.506+0000 7f9e90094640 1 -- 172.21.15.110:0/1213138913 shutdown_connections 2024-09-15T03:58:49.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.506+0000 7f9e90094640 1 --2- 172.21.15.110:0/1213138913 >> v2:172.21.15.177:6812/28201 conn(0x7f9e6c038700 0x7f9e6c03abb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.506+0000 7f9e90094640 1 --2- 172.21.15.110:0/1213138913 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e8809e6c0 0x7f9e88119200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.506+0000 7f9e90094640 1 --2- 172.21.15.110:0/1213138913 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9e8809e2f0 0x7f9e8812f700 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.506+0000 7f9e90094640 1 --2- 172.21.15.110:0/1213138913 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e8806e590 0x7f9e8813eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.506+0000 7f9e90094640 1 -- 172.21.15.110:0/1213138913 >> 172.21.15.110:0/1213138913 conn(0x7f9e880639f0 msgr2=0x7f9e880a17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:49.507 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.506+0000 7f9e90094640 1 -- 172.21.15.110:0/1213138913 shutdown_connections 2024-09-15T03:58:49.508 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.506+0000 7f9e90094640 1 -- 172.21.15.110:0/1213138913 wait complete. 2024-09-15T03:58:49.517 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T03:58:36.398831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-15T03:58:41.004777+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2024-09-15T03:58:49.518 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-15T03:58:49.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.611+0000 7f3e4c554640 10 monclient: get_monmap_and_config 2024-09-15T03:58:49.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.611+0000 7f3e4c554640 10 monclient: build_initial_monmap 2024-09-15T03:58:49.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.611+0000 7f3e4c554640 10 monclient: monmap: 2024-09-15T03:58:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:49.612697+0000 2024-09-15T03:58:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:49.612697+0000 2024-09-15T03:58:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:49.613 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:49.613 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 1 Processor -- start 2024-09-15T03:58:49.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 1 -- start start 2024-09-15T03:58:49.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 10 monclient: init 2024-09-15T03:58:49.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:49.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-15T03:58:49.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e44131630 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3e4406e5a0 0x7f3e4406e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 10 monclient(hunting): picked mon.noname-c con 0x7f3e4406e5a0 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e4406eeb0 0x7f3e44138970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 10 monclient(hunting): picked mon.noname-b con 0x7f3e4406eeb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4409f350 con 0x7f3e44131630 2024-09-15T03:58:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e44068390 con 0x7f3e4406e5a0 2024-09-15T03:58:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e44069570 con 0x7f3e4406eeb0 2024-09-15T03:58:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.612+0000 7f3e4c554640 10 monclient(hunting): authenticate will time out at 753.256966s 2024-09-15T03:58:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4a2f0640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3e44131630 0x7f3e44131a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:34136/0 (socket says 172.21.15.110:34136) 2024-09-15T03:58:49.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e4406eeb0 0x7f3e44138970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4a2f0640 1 -- 172.21.15.110:0/3391814351 learned_addr learned my addr 172.21.15.110:0/3391814351 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 10 monclient(hunting): get_auth_request con 0x7f3e4406eeb0 auth_method 0 2024-09-15T03:58:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 10 monclient(hunting): handle_auth_done global_id 4337 payload 274 2024-09-15T03:58:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 1 -- 172.21.15.110:0/3391814351 >> v2:172.21.15.177:3300/0 conn(0x7f3e4406e5a0 msgr2=0x7f3e4406e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e49aef640 1 --2- 172.21.15.110:0/3391814351 >> v2:172.21.15.177:3300/0 conn(0x7f3e4406e5a0 0x7f3e4406e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 1 --2- 172.21.15.110:0/3391814351 >> v2:172.21.15.177:3300/0 conn(0x7f3e4406e5a0 0x7f3e4406e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 1 -- 172.21.15.110:0/3391814351 >> v1:172.21.15.110:6789/0 conn(0x7f3e44131630 legacy=0x7f3e44131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:49.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 10 monclient: _finish_hunting 0 2024-09-15T03:58:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 1 monclient: found mon.noname-b 2024-09-15T03:58:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e4aaf1640 1 -- 172.21.15.110:0/3391814351 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e44069850 con 0x7f3e4406eeb0 2024-09-15T03:58:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e49aef640 10 monclient: get_auth_request con 0x7f3e4406e5a0 auth_method 0 2024-09-15T03:58:49.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.613+0000 7f3e49aef640 1 --2- 172.21.15.110:0/3391814351 >> v2:172.21.15.177:3300/0 conn(0x7f3e4406e5a0 0x7f3e4406e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:58:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4aaf1640 1 -- 172.21.15.110:0/3391814351 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e4406eeb0 msgr2=0x7f3e44138970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4aaf1640 1 --2- 172.21.15.110:0/3391814351 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e4406eeb0 0x7f3e44138970 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009e60 tx=0x7f3e3c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2489441ccd88ee83 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e492ee640 1 -- 172.21.15.110:0/3391814351 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c007840 con 0x7f3e4406eeb0 2024-09-15T03:58:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e492ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e492ee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e492ee640 10 monclient: dump: 2024-09-15T03:58:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.625 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.626 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e492ee640 10 monclient: _finish_auth 0 2024-09-15T03:58:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e492ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e492ee640 1 -- 172.21.15.110:0/3391814351 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c02d050 con 0x7f3e4406eeb0 2024-09-15T03:58:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4c554640 5 monclient: authenticate success, global_id 4337 2024-09-15T03:58:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4c554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e492ee640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4c554640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:49.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e492ee640 1 -- 172.21.15.110:0/3391814351 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c007c10 con 0x7f3e4406eeb0 2024-09-15T03:58:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4c554640 4 set_mon_vals no callback set 2024-09-15T03:58:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e492ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e492ee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e492ee640 10 monclient: dump: 2024-09-15T03:58:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.627 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.628 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4c554640 10 monclient: shutdown 2024-09-15T03:58:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4c554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4c554640 1 -- 172.21.15.110:0/3391814351 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e4406eeb0 msgr2=0x7f3e44138970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4c554640 1 --2- 172.21.15.110:0/3391814351 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e4406eeb0 0x7f3e44138970 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009e60 tx=0x7f3e3c009780 comp rx=0 tx=0).stop 2024-09-15T03:58:49.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4c554640 1 -- 172.21.15.110:0/3391814351 shutdown_connections 2024-09-15T03:58:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4c554640 1 --2- 172.21.15.110:0/3391814351 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e4406eeb0 0x7f3e44138970 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4c554640 1 --2- 172.21.15.110:0/3391814351 >> v2:172.21.15.177:3300/0 conn(0x7f3e4406e5a0 0x7f3e4406e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4c554640 1 -- 172.21.15.110:0/3391814351 >> 172.21.15.110:0/3391814351 conn(0x7f3e440639f0 msgr2=0x7f3e440a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.616+0000 7f3e4c554640 1 -- 172.21.15.110:0/3391814351 shutdown_connections 2024-09-15T03:58:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.617+0000 7f3e4c554640 1 -- 172.21.15.110:0/3391814351 wait complete. 2024-09-15T03:58:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.617+0000 7f3e4c554640 10 monclient: build_initial_monmap 2024-09-15T03:58:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.617+0000 7f3e4c554640 10 monclient: monmap: 2024-09-15T03:58:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:49.629 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.630 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:49.630 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:49.630 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:49.630 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.617+0000 7f3e4c554640 1 Processor -- start 2024-09-15T03:58:49.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.617+0000 7f3e4c554640 1 -- start start 2024-09-15T03:58:49.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.617+0000 7f3e4c554640 10 monclient: init 2024-09-15T03:58:49.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:49.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:58:49.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e4406e5a0 0x7f3e44135db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 10 monclient(hunting): picked mon.noname-a con 0x7f3e4406e5a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:49.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e4406eeb0 0x7f3e441362f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 10 monclient(hunting): picked mon.noname-b con 0x7f3e4406eeb0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:49.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e44131630 0x7f3e44136830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 10 monclient(hunting): picked mon.noname-c con 0x7f3e44131630 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:49.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4411a7c0 con 0x7f3e4406e5a0 2024-09-15T03:58:49.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e44138190 con 0x7f3e4406eeb0 2024-09-15T03:58:49.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e441191e0 con 0x7f3e44131630 2024-09-15T03:58:49.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:49.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4c554640 10 monclient(hunting): authenticate will time out at 753.262332s 2024-09-15T03:58:49.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4a2f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e4406e5a0 0x7f3e44135db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e44131630 0x7f3e44136830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4a2f0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e4406e5a0 0x7f3e44135db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53096/0 (socket says 172.21.15.110:53096) 2024-09-15T03:58:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 10 monclient(hunting): get_auth_request con 0x7f3e44131630 auth_method 0 2024-09-15T03:58:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4a2f0640 1 -- 172.21.15.110:0/957409746 learned_addr learned my addr 172.21.15.110:0/957409746 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4a2f0640 10 monclient(hunting): get_auth_request con 0x7f3e4406e5a0 auth_method 0 2024-09-15T03:58:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4a2f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4a2f0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4a2f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4a2f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4a2f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4a2f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 10 monclient(hunting): handle_auth_done global_id 4340 payload 824 2024-09-15T03:58:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 1 -- 172.21.15.110:0/957409746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e4406eeb0 msgr2=0x7f3e441362f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 1 --2- 172.21.15.110:0/957409746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e4406eeb0 0x7f3e441362f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 1 -- 172.21.15.110:0/957409746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e4406e5a0 msgr2=0x7f3e44135db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 1 --2- 172.21.15.110:0/957409746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e4406e5a0 0x7f3e44135db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 10 monclient: _finish_hunting 0 2024-09-15T03:58:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 1 monclient: found mon.noname-c 2024-09-15T03:58:49.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4aaf1640 1 -- 172.21.15.110:0/957409746 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c009a20 con 0x7f3e44131630 2024-09-15T03:58:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.618+0000 7f3e4a2f0640 1 --2- 172.21.15.110:0/957409746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e4406e5a0 0x7f3e44135db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.620+0000 7f3e4aaf1640 1 -- 172.21.15.110:0/957409746 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e44131630 msgr2=0x7f3e44136830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.620+0000 7f3e4aaf1640 1 --2- 172.21.15.110:0/957409746 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e44131630 0x7f3e44136830 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c031000 tx=0x7f3e3c037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e3a7fc640 1 -- 172.21.15.110:0/957409746 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c031cd0 con 0x7f3e44131630 2024-09-15T03:58:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e3a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e3a7fc640 10 monclient: dump: 2024-09-15T03:58:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.634 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.635 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e3a7fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e3a7fc640 1 -- 172.21.15.110:0/957409746 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c040070 con 0x7f3e44131630 2024-09-15T03:58:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e4c554640 5 monclient: authenticate success, global_id 4340 2024-09-15T03:58:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:49.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e4c554640 10 monclient: _renew_subs 2024-09-15T03:58:49.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e3a7fc640 1 -- 172.21.15.110:0/957409746 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c045680 con 0x7f3e44131630 2024-09-15T03:58:49.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e4c554640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e4c554640 1 -- 172.21.15.110:0/957409746 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e441379d0 con 0x7f3e44131630 2024-09-15T03:58:49.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e48aed640 4 set_mon_vals no callback set 2024-09-15T03:58:49.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:49.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e3a7fc640 10 monclient: dump: 2024-09-15T03:58:49.636 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.636 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.636 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.636 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.636 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.637 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e4c554640 10 monclient: _renew_subs 2024-09-15T03:58:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e4c554640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e4c554640 1 -- 172.21.15.110:0/957409746 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e44138c10 con 0x7f3e44131630 2024-09-15T03:58:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e3a7fc640 1 -- 172.21.15.110:0/957409746 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f3e3c042030 con 0x7f3e44131630 2024-09-15T03:58:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.621+0000 7f3e4c554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.622+0000 7f3e4c554640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.622+0000 7f3e4c554640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.622+0000 7f3e4c554640 1 -- 172.21.15.110:0/957409746 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e4411a120 con 0x7f3e44131630 2024-09-15T03:58:49.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.622+0000 7f3e3a7fc640 1 --2- 172.21.15.110:0/957409746 >> v2:172.21.15.177:6812/28201 conn(0x7f3e28038700 0x7f3e2803abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.622+0000 7f3e3a7fc640 1 -- 172.21.15.110:0/957409746 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f3e3c06d0c0 con 0x7f3e44131630 2024-09-15T03:58:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.622+0000 7f3e4a2f0640 1 --2- 172.21.15.110:0/957409746 >> v2:172.21.15.177:6812/28201 conn(0x7f3e28038700 0x7f3e2803abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.622+0000 7f3e4a2f0640 10 monclient: get_auth_request con 0x7f3e28038700 auth_method 0 2024-09-15T03:58:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.625+0000 7f3e4a2f0640 1 --2- 172.21.15.110:0/957409746 >> v2:172.21.15.177:6812/28201 conn(0x7f3e28038700 0x7f3e2803abb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3e40006fd0 tx=0x7f3e40004960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.626+0000 7f3e3a7fc640 1 -- 172.21.15.110:0/957409746 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f3e3c031e30 con 0x7f3e44131630 2024-09-15T03:58:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.626+0000 7f3e3a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:49.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.626+0000 7f3e3a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:49.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.779+0000 7f3e4c554640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T03:58:49.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.779+0000 7f3e4c554640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T03:58:49.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.779+0000 7f3e4c554640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:49.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.779+0000 7f3e4c554640 1 -- 172.21.15.110:0/957409746 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3e4406e240 con 0x7f3e44131630 2024-09-15T03:58:49.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.779+0000 7f3e3a7fc640 1 -- 172.21.15.110:0/957409746 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3e3c0500c0 con 0x7f3e44131630 2024-09-15T03:58:49.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.779+0000 7f3e3a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T03:58:49.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.779+0000 7f3e3a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:49.782 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 1 2024-09-15T03:58:49.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.781+0000 7f3e4c554640 1 -- 172.21.15.110:0/957409746 >> v2:172.21.15.177:6812/28201 conn(0x7f3e28038700 msgr2=0x7f3e2803abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.781+0000 7f3e4c554640 1 --2- 172.21.15.110:0/957409746 >> v2:172.21.15.177:6812/28201 conn(0x7f3e28038700 0x7f3e2803abb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3e40006fd0 tx=0x7f3e40004960 comp rx=0 tx=0).stop 2024-09-15T03:58:49.784 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.781+0000 7f3e4c554640 10 monclient: shutdown 2024-09-15T03:58:49.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.781+0000 7f3e4c554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:49.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.781+0000 7f3e4c554640 1 -- 172.21.15.110:0/957409746 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e44131630 msgr2=0x7f3e44136830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.781+0000 7f3e4c554640 1 --2- 172.21.15.110:0/957409746 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e44131630 0x7f3e44136830 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c031000 tx=0x7f3e3c037000 comp rx=0 tx=0).stop 2024-09-15T03:58:49.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.781+0000 7f3e4c554640 1 -- 172.21.15.110:0/957409746 shutdown_connections 2024-09-15T03:58:49.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.781+0000 7f3e4c554640 1 --2- 172.21.15.110:0/957409746 >> v2:172.21.15.177:6812/28201 conn(0x7f3e28038700 0x7f3e2803abb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.781+0000 7f3e4c554640 1 --2- 172.21.15.110:0/957409746 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3e44131630 0x7f3e44136830 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.781+0000 7f3e4c554640 1 --2- 172.21.15.110:0/957409746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e4406eeb0 0x7f3e441362f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.781+0000 7f3e4c554640 1 --2- 172.21.15.110:0/957409746 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e4406e5a0 0x7f3e44135db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.781+0000 7f3e4c554640 1 -- 172.21.15.110:0/957409746 >> 172.21.15.110:0/957409746 conn(0x7f3e440639f0 msgr2=0x7f3e440a10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:49.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.781+0000 7f3e4c554640 1 -- 172.21.15.110:0/957409746 shutdown_connections 2024-09-15T03:58:49.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.781+0000 7f3e4c554640 1 -- 172.21.15.110:0/957409746 wait complete. 2024-09-15T03:58:49.793 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-15T03:58:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.887+0000 7f928f42b640 10 monclient: get_monmap_and_config 2024-09-15T03:58:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.887+0000 7f928f42b640 10 monclient: build_initial_monmap 2024-09-15T03:58:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.887+0000 7f928f42b640 10 monclient: monmap: 2024-09-15T03:58:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:49.888431+0000 2024-09-15T03:58:49.888 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:49.888431+0000 2024-09-15T03:58:49.889 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:49.889 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.889 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:49.889 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:49.889 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:49.889 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:49.889 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:49.889 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.887+0000 7f928f42b640 1 Processor -- start 2024-09-15T03:58:49.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 1 -- start start 2024-09-15T03:58:49.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 10 monclient: init 2024-09-15T03:58:49.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:49.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-15T03:58:49.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f92880a1bf0 0x7f9288131810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 10 monclient(hunting): picked mon.noname-a con 0x7f92880a1bf0 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:49.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f928809e780 0x7f928809eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 10 monclient(hunting): picked mon.noname-b con 0x7f928809e780 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:49.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f928809f760 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:49.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f928809c9b0 con 0x7f928809f760 2024-09-15T03:58:49.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92880a0c20 con 0x7f92880a1bf0 2024-09-15T03:58:49.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f928809cc90 con 0x7f928809e780 2024-09-15T03:58:49.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:49.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928f42b640 10 monclient(hunting): authenticate will time out at 753.532699s 2024-09-15T03:58:49.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928d1c7640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f92880a1bf0 0x7f9288131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928c9c6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f928809e780 0x7f928809eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928d9c8640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f928809f760 0x7f928809fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:34140/0 (socket says 172.21.15.110:34140) 2024-09-15T03:58:49.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928d9c8640 1 -- 172.21.15.110:0/1789377460 learned_addr learned my addr 172.21.15.110:0/1789377460 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:49.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928d1c7640 1 --2- 172.21.15.110:0/1789377460 >> v2:172.21.15.110:3300/0 conn(0x7f92880a1bf0 0x7f9288131810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53100/0 (socket says 172.21.15.110:53100) 2024-09-15T03:58:49.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928c9c6640 10 monclient(hunting): get_auth_request con 0x7f928809e780 auth_method 0 2024-09-15T03:58:49.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928c9c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928c9c6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928c9c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928d1c7640 10 monclient(hunting): get_auth_request con 0x7f92880a1bf0 auth_method 0 2024-09-15T03:58:49.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928d1c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928d1c7640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928d1c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928c9c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:49.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928c9c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:49.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928c9c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:49.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928d1c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:49.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928d1c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:49.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.888+0000 7f928d1c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:49.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.889+0000 7f928d1c7640 10 monclient(hunting): handle_auth_done global_id 4314 payload 274 2024-09-15T03:58:49.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.889+0000 7f928d1c7640 1 -- 172.21.15.110:0/1789377460 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f928809e780 msgr2=0x7f928809eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:49.894 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.889+0000 7f928d1c7640 1 --2- 172.21.15.110:0/1789377460 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f928809e780 0x7f928809eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.889+0000 7f928d1c7640 1 -- 172.21.15.110:0/1789377460 >> v1:172.21.15.110:6789/0 conn(0x7f928809f760 legacy=0x7f928809fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:49.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.889+0000 7f928d1c7640 10 monclient: _finish_hunting 0 2024-09-15T03:58:49.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.889+0000 7f928d1c7640 1 monclient: found mon.noname-a 2024-09-15T03:58:49.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.889+0000 7f928d1c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:49.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.889+0000 7f928d1c7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:49.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.889+0000 7f928d1c7640 1 -- 172.21.15.110:0/1789377460 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92880a1fc0 con 0x7f92880a1bf0 2024-09-15T03:58:49.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.889+0000 7f928c9c6640 1 --2- 172.21.15.110:0/1789377460 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f928809e780 0x7f928809eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:49.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.891+0000 7f928d1c7640 1 -- 172.21.15.110:0/1789377460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92880a1bf0 msgr2=0x7f9288131810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:49.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.891+0000 7f928d1c7640 1 --2- 172.21.15.110:0/1789377460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92880a1bf0 0x7f9288131810 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f927c009850 tx=0x7f927c0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc1eed22347e04 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:49.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f927bfff640 1 -- 172.21.15.110:0/1789377460 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c02ae80 con 0x7f92880a1bf0 2024-09-15T03:58:49.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f927bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f927bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:49.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f927bfff640 10 monclient: dump: 2024-09-15T03:58:49.901 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.901 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.901 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.901 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.901 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.901 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.901 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.901 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.901 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f927bfff640 10 monclient: _finish_auth 0 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f927bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f927bfff640 1 -- 172.21.15.110:0/1789377460 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f927c032440 con 0x7f92880a1bf0 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f928f42b640 5 monclient: authenticate success, global_id 4314 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f928f42b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f927bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f927bfff640 1 -- 172.21.15.110:0/1789377460 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927c0327c0 con 0x7f92880a1bf0 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f928f42b640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f928f42b640 4 set_mon_vals no callback set 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f927bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f927bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f927bfff640 10 monclient: dump: 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.902 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.903 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.903 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.903 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.903 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.903 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.903 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.903 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.903 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f928f42b640 10 monclient: shutdown 2024-09-15T03:58:49.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f928f42b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:49.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f928f42b640 1 -- 172.21.15.110:0/1789377460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92880a1bf0 msgr2=0x7f9288131810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f928f42b640 1 --2- 172.21.15.110:0/1789377460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92880a1bf0 0x7f9288131810 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f927c009850 tx=0x7f927c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f928f42b640 1 -- 172.21.15.110:0/1789377460 shutdown_connections 2024-09-15T03:58:49.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f928f42b640 1 --2- 172.21.15.110:0/1789377460 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f928809e780 0x7f928809eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f928f42b640 1 --2- 172.21.15.110:0/1789377460 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92880a1bf0 0x7f9288131810 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f928f42b640 1 -- 172.21.15.110:0/1789377460 >> 172.21.15.110:0/1789377460 conn(0x7f9288062dd0 msgr2=0x7f92880631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:49.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f928f42b640 1 -- 172.21.15.110:0/1789377460 shutdown_connections 2024-09-15T03:58:49.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.892+0000 7f928f42b640 1 -- 172.21.15.110:0/1789377460 wait complete. 2024-09-15T03:58:49.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.893+0000 7f928f42b640 10 monclient: build_initial_monmap 2024-09-15T03:58:49.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.893+0000 7f928f42b640 10 monclient: monmap: 2024-09-15T03:58:49.904 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:49.904 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:49.904 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:49.904 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:49.904 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:49.904 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.904 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:49.904 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:49.905 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:49.905 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.893+0000 7f928f42b640 1 Processor -- start 2024-09-15T03:58:49.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.893+0000 7f928f42b640 1 -- start start 2024-09-15T03:58:49.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.893+0000 7f928f42b640 10 monclient: init 2024-09-15T03:58:49.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:49.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:58:49.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f928809e780 0x7f9288099b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 10 monclient(hunting): picked mon.noname-b con 0x7f928809e780 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:49.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f928809f760 0x7f92881307f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 10 monclient(hunting): picked mon.noname-c con 0x7f928809f760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:49.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92880a1bf0 0x7f9288130d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 10 monclient(hunting): picked mon.noname-a con 0x7f92880a1bf0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:49.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928809a7c0 con 0x7f92880a1bf0 2024-09-15T03:58:49.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928809c250 con 0x7f928809e780 2024-09-15T03:58:49.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:49.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f928809ae00 con 0x7f928809f760 2024-09-15T03:58:49.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:49.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928f42b640 10 monclient(hunting): authenticate will time out at 753.538217s 2024-09-15T03:58:49.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92880a1bf0 0x7f9288130d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928c9c6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f928809f760 0x7f92881307f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92880a1bf0 0x7f9288130d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53102/0 (socket says 172.21.15.110:53102) 2024-09-15T03:58:49.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 1 -- 172.21.15.110:0/4265482042 learned_addr learned my addr 172.21.15.110:0/4265482042 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:49.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 10 monclient(hunting): get_auth_request con 0x7f92880a1bf0 auth_method 0 2024-09-15T03:58:49.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928c9c6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f928809f760 0x7f92881307f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43488/0 (socket says 172.21.15.110:43488) 2024-09-15T03:58:49.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928c9c6640 10 monclient(hunting): get_auth_request con 0x7f928809f760 auth_method 0 2024-09-15T03:58:49.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928c9c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:49.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928c9c6640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:49.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928c9c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:49.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:49.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:49.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:49.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928c9c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:49.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928c9c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:49.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928c9c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:49.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d1c7640 1 --2- 172.21.15.110:0/4265482042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f928809e780 0x7f9288099b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 10 monclient(hunting): handle_auth_done global_id 4317 payload 824 2024-09-15T03:58:49.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 1 -- 172.21.15.110:0/4265482042 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f928809f760 msgr2=0x7f92881307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 1 --2- 172.21.15.110:0/4265482042 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f928809f760 0x7f92881307f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 1 -- 172.21.15.110:0/4265482042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f928809e780 msgr2=0x7f9288099b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:49.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 1 --2- 172.21.15.110:0/4265482042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f928809e780 0x7f9288099b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:49.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 10 monclient: _finish_hunting 0 2024-09-15T03:58:49.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 1 monclient: found mon.noname-a 2024-09-15T03:58:49.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:49.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:49.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928d9c8640 1 -- 172.21.15.110:0/4265482042 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9284009390 con 0x7f92880a1bf0 2024-09-15T03:58:49.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.894+0000 7f928c9c6640 1 --2- 172.21.15.110:0/4265482042 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f928809f760 0x7f92881307f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:49.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.896+0000 7f928d9c8640 1 -- 172.21.15.110:0/4265482042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92880a1bf0 msgr2=0x7f9288130d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:49.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.896+0000 7f928d9c8640 1 --2- 172.21.15.110:0/4265482042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92880a1bf0 0x7f9288130d30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f928000ccf0 tx=0x7f9280005370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:49.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.896+0000 7f92797fa640 1 -- 172.21.15.110:0/4265482042 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928000aa60 con 0x7f92880a1bf0 2024-09-15T03:58:49.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.896+0000 7f92797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.896+0000 7f92797fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:49.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.896+0000 7f92797fa640 10 monclient: dump: 2024-09-15T03:58:49.909 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.909 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.909 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.909 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.909 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.910 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.910 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.910 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.910 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.910 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.896+0000 7f92797fa640 10 monclient: _finish_auth 0 2024-09-15T03:58:49.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.896+0000 7f92797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:49.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.896+0000 7f928f42b640 5 monclient: authenticate success, global_id 4317 2024-09-15T03:58:49.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.896+0000 7f92797fa640 1 -- 172.21.15.110:0/4265482042 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928000abe0 con 0x7f92880a1bf0 2024-09-15T03:58:49.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f928f42b640 10 monclient: _renew_subs 2024-09-15T03:58:49.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f928f42b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:49.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f928f42b640 1 -- 172.21.15.110:0/4265482042 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927c009390 con 0x7f92880a1bf0 2024-09-15T03:58:49.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f92797fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f92797fa640 1 -- 172.21.15.110:0/4265482042 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92800070f0 con 0x7f92880a1bf0 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f92797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f92797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f92797fa640 10 monclient: dump: 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:49.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f927b7fe640 4 set_mon_vals no callback set 2024-09-15T03:58:49.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f928f42b640 10 monclient: _renew_subs 2024-09-15T03:58:49.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f928f42b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:49.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f928f42b640 1 -- 172.21.15.110:0/4265482042 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928811c1e0 con 0x7f92880a1bf0 2024-09-15T03:58:49.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f92797fa640 1 -- 172.21.15.110:0/4265482042 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f92800074a0 con 0x7f92880a1bf0 2024-09-15T03:58:49.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f928f42b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:49.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f928f42b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:49.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f928f42b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:49.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.897+0000 7f928f42b640 1 -- 172.21.15.110:0/4265482042 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f928811e0a0 con 0x7f92880a1bf0 2024-09-15T03:58:49.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.898+0000 7f92797fa640 1 --2- 172.21.15.110:0/4265482042 >> v2:172.21.15.177:6812/28201 conn(0x7f926c038700 0x7f926c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:49.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.898+0000 7f92797fa640 1 -- 172.21.15.110:0/4265482042 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f928004b720 con 0x7f92880a1bf0 2024-09-15T03:58:49.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.898+0000 7f928d1c7640 1 --2- 172.21.15.110:0/4265482042 >> v2:172.21.15.177:6812/28201 conn(0x7f926c038700 0x7f926c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:49.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.898+0000 7f928d1c7640 10 monclient: get_auth_request con 0x7f926c038700 auth_method 0 2024-09-15T03:58:49.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.900+0000 7f928d1c7640 1 --2- 172.21.15.110:0/4265482042 >> v2:172.21.15.177:6812/28201 conn(0x7f926c038700 0x7f926c03abb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f92880a2c10 tx=0x7f927c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:49.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.902+0000 7f92797fa640 1 -- 172.21.15.110:0/4265482042 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f928811e0a0 con 0x7f92880a1bf0 2024-09-15T03:58:49.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.902+0000 7f92797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:49.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:49.902+0000 7f92797fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:50.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.054+0000 7f928f42b640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T03:58:50.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.054+0000 7f928f42b640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T03:58:50.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.054+0000 7f928f42b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:50.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.054+0000 7f928f42b640 1 -- 172.21.15.110:0/4265482042 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f928811e0a0 con 0x7f92880a1bf0 2024-09-15T03:58:50.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.055+0000 7f92797fa640 1 -- 172.21.15.110:0/4265482042 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f928811e0a0 con 0x7f92880a1bf0 2024-09-15T03:58:50.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.055+0000 7f92797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T03:58:50.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.055+0000 7f92797fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:50.056 INFO:teuthology.orchestra.run.smithi110.stdout:pg_num: 8 2024-09-15T03:58:50.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.056+0000 7f928f42b640 1 -- 172.21.15.110:0/4265482042 >> v2:172.21.15.177:6812/28201 conn(0x7f926c038700 msgr2=0x7f926c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.056+0000 7f928f42b640 1 --2- 172.21.15.110:0/4265482042 >> v2:172.21.15.177:6812/28201 conn(0x7f926c038700 0x7f926c03abb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f92880a2c10 tx=0x7f927c032000 comp rx=0 tx=0).stop 2024-09-15T03:58:50.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.056+0000 7f928f42b640 10 monclient: shutdown 2024-09-15T03:58:50.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.056+0000 7f928f42b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:50.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.056+0000 7f928f42b640 1 -- 172.21.15.110:0/4265482042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92880a1bf0 msgr2=0x7f9288130d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.056+0000 7f928f42b640 1 --2- 172.21.15.110:0/4265482042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92880a1bf0 0x7f9288130d30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f928000ccf0 tx=0x7f9280005370 comp rx=0 tx=0).stop 2024-09-15T03:58:50.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.056+0000 7f928f42b640 1 -- 172.21.15.110:0/4265482042 shutdown_connections 2024-09-15T03:58:50.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.056+0000 7f928f42b640 1 --2- 172.21.15.110:0/4265482042 >> v2:172.21.15.177:6812/28201 conn(0x7f926c038700 0x7f926c03abb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.056+0000 7f928f42b640 1 --2- 172.21.15.110:0/4265482042 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92880a1bf0 0x7f9288130d30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.056+0000 7f928f42b640 1 --2- 172.21.15.110:0/4265482042 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f928809f760 0x7f92881307f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.056+0000 7f928f42b640 1 --2- 172.21.15.110:0/4265482042 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f928809e780 0x7f9288099b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.056+0000 7f928f42b640 1 -- 172.21.15.110:0/4265482042 >> 172.21.15.110:0/4265482042 conn(0x7f9288062dd0 msgr2=0x7f92880a4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:50.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.057+0000 7f928f42b640 1 -- 172.21.15.110:0/4265482042 shutdown_connections 2024-09-15T03:58:50.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.057+0000 7f928f42b640 1 -- 172.21.15.110:0/4265482042 wait complete. 2024-09-15T03:58:50.067 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-15T03:58:50.067 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:58:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.160+0000 7fad5ff1d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.160+0000 7fad5ff1d640 10 monclient: build_initial_monmap 2024-09-15T03:58:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.160+0000 7fad5ff1d640 10 monclient: monmap: 2024-09-15T03:58:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:50.161156+0000 2024-09-15T03:58:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:50.161156+0000 2024-09-15T03:58:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:50.161 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:50.162 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:50.162 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:50.162 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:50.162 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.160+0000 7fad5ff1d640 1 Processor -- start 2024-09-15T03:58:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.160+0000 7fad5ff1d640 1 -- start start 2024-09-15T03:58:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 10 monclient: init 2024-09-15T03:58:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-15T03:58:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fad580a1a60 0x7fad5809d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 10 monclient(hunting): picked mon.noname-a con 0x7fad580a1a60 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad5809f870 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fad580a0380 0x7fad58138b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.164 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 10 monclient(hunting): picked mon.noname-c con 0x7fad580a0380 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad580a1260 con 0x7fad5809f870 2024-09-15T03:58:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5809cdf0 con 0x7fad580a1a60 2024-09-15T03:58:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5809c870 con 0x7fad580a0380 2024-09-15T03:58:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5ff1d640 10 monclient(hunting): authenticate will time out at 753.805547s 2024-09-15T03:58:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fad580a1a60 0x7fad5809d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fad580a1a60 0x7fad5809d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53112/0 (socket says 172.21.15.110:53112) 2024-09-15T03:58:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 1 -- 172.21.15.110:0/3443220320 learned_addr learned my addr 172.21.15.110:0/3443220320 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 10 monclient(hunting): get_auth_request con 0x7fad580a1a60 auth_method 0 2024-09-15T03:58:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:50.165 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 10 monclient(hunting): handle_auth_done global_id 4320 payload 274 2024-09-15T03:58:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 1 -- 172.21.15.110:0/3443220320 >> v2:172.21.15.177:3300/0 conn(0x7fad580a0380 msgr2=0x7fad58138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 1 --2- 172.21.15.110:0/3443220320 >> v2:172.21.15.177:3300/0 conn(0x7fad580a0380 0x7fad58138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 1 -- 172.21.15.110:0/3443220320 >> v1:172.21.15.177:6789/0 conn(0x7fad5809f870 legacy=0x7fad5809fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 10 monclient: _finish_hunting 0 2024-09-15T03:58:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 1 monclient: found mon.noname-a 2024-09-15T03:58:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:50.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.161+0000 7fad5dcb9640 1 -- 172.21.15.110:0/3443220320 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad580a0c60 con 0x7fad580a1a60 2024-09-15T03:58:50.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.164+0000 7fad5dcb9640 1 -- 172.21.15.110:0/3443220320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad580a1a60 msgr2=0x7fad5809d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:50.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.164+0000 7fad5dcb9640 1 --2- 172.21.15.110:0/3443220320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad580a1a60 0x7fad5809d4f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fad580a0a10 tx=0x7fad540091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d8623f56d3ef66 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:50.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.164+0000 7fad5ccb7640 1 -- 172.21.15.110:0/3443220320 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54032440 con 0x7fad580a1a60 2024-09-15T03:58:50.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.164+0000 7fad5ccb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:50.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ccb7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:50.172 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ccb7640 10 monclient: dump: 2024-09-15T03:58:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:50.173 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ccb7640 10 monclient: _finish_auth 0 2024-09-15T03:58:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ccb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:50.173 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ff1d640 5 monclient: authenticate success, global_id 4320 2024-09-15T03:58:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ff1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ccb7640 1 -- 172.21.15.110:0/3443220320 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad540325a0 con 0x7fad580a1a60 2024-09-15T03:58:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ccb7640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ff1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ccb7640 1 -- 172.21.15.110:0/3443220320 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad54032940 con 0x7fad580a1a60 2024-09-15T03:58:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ccb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ff1d640 4 set_mon_vals no callback set 2024-09-15T03:58:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ccb7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ccb7640 10 monclient: dump: 2024-09-15T03:58:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.174 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:50.175 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.175 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:50.175 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:50.175 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:50.175 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ff1d640 10 monclient: shutdown 2024-09-15T03:58:50.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ff1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:50.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ff1d640 1 -- 172.21.15.110:0/3443220320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad580a1a60 msgr2=0x7fad5809d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.175 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ff1d640 1 --2- 172.21.15.110:0/3443220320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad580a1a60 0x7fad5809d4f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fad580a0a10 tx=0x7fad540091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ff1d640 1 -- 172.21.15.110:0/3443220320 shutdown_connections 2024-09-15T03:58:50.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ff1d640 1 --2- 172.21.15.110:0/3443220320 >> v2:172.21.15.177:3300/0 conn(0x7fad580a0380 0x7fad58138b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ff1d640 1 --2- 172.21.15.110:0/3443220320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad580a1a60 0x7fad5809d4f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ff1d640 1 -- 172.21.15.110:0/3443220320 >> 172.21.15.110:0/3443220320 conn(0x7fad58063270 msgr2=0x7fad58063670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:50.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ff1d640 1 -- 172.21.15.110:0/3443220320 shutdown_connections 2024-09-15T03:58:50.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.165+0000 7fad5ff1d640 1 -- 172.21.15.110:0/3443220320 wait complete. 2024-09-15T03:58:50.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 10 monclient: build_initial_monmap 2024-09-15T03:58:50.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 10 monclient: monmap: 2024-09-15T03:58:50.176 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:50.176 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:50.176 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:50.176 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 1 Processor -- start 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 1 -- start start 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 10 monclient: init 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad5809f870 0x7fad58099660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 10 monclient(hunting): picked mon.noname-a con 0x7fad5809f870 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad580a0380 0x7fad58099ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 10 monclient(hunting): picked mon.noname-b con 0x7fad580a0380 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:50.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad580a1a60 0x7fad5809a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 10 monclient(hunting): picked mon.noname-c con 0x7fad580a1a60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:50.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:50.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5811da50 con 0x7fad5809f870 2024-09-15T03:58:50.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:50.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad580a1450 con 0x7fad580a0380 2024-09-15T03:58:50.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:50.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5811c490 con 0x7fad580a1a60 2024-09-15T03:58:50.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.166+0000 7fad5ff1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:50.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5ff1d640 10 monclient(hunting): authenticate will time out at 753.811009s 2024-09-15T03:58:50.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad5809f870 0x7fad58099660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad5809f870 0x7fad58099660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53122/0 (socket says 172.21.15.110:53122) 2024-09-15T03:58:50.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 1 -- 172.21.15.110:0/2608809404 learned_addr learned my addr 172.21.15.110:0/2608809404 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:50.178 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 10 monclient(hunting): get_auth_request con 0x7fad5809f870 auth_method 0 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5e4ba640 1 --2- 172.21.15.110:0/2608809404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad580a1a60 0x7fad5809a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5e4ba640 10 monclient(hunting): get_auth_request con 0x7fad580a1a60 auth_method 0 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5e4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5e4ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5e4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5e4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5e4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5e4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5d4b8640 1 --2- 172.21.15.110:0/2608809404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad580a0380 0x7fad58099ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.179 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 10 monclient(hunting): handle_auth_done global_id 4323 payload 824 2024-09-15T03:58:50.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 1 -- 172.21.15.110:0/2608809404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad580a1a60 msgr2=0x7fad5809a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 1 --2- 172.21.15.110:0/2608809404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad580a1a60 0x7fad5809a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 1 -- 172.21.15.110:0/2608809404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad580a0380 msgr2=0x7fad58099ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 1 --2- 172.21.15.110:0/2608809404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad580a0380 0x7fad58099ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 10 monclient: _finish_hunting 0 2024-09-15T03:58:50.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 1 monclient: found mon.noname-a 2024-09-15T03:58:50.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:50.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:50.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5dcb9640 1 -- 172.21.15.110:0/2608809404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad54009450 con 0x7fad5809f870 2024-09-15T03:58:50.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5d4b8640 10 monclient: get_auth_request con 0x7fad580a0380 auth_method 0 2024-09-15T03:58:50.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.167+0000 7fad5d4b8640 1 --2- 172.21.15.110:0/2608809404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad580a0380 0x7fad58099ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:58:50.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad5dcb9640 1 -- 172.21.15.110:0/2608809404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad5809f870 msgr2=0x7fad58099660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:50.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad5dcb9640 1 --2- 172.21.15.110:0/2608809404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad5809f870 0x7fad58099660 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fad54035040 tx=0x7fad54032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:50.180 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad4dffb640 1 -- 172.21.15.110:0/2608809404 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5403b8f0 con 0x7fad5809f870 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad4dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad4dffb640 10 monclient: dump: 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad4dffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:50.181 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad5ff1d640 5 monclient: authenticate success, global_id 4323 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad4dffb640 1 -- 172.21.15.110:0/2608809404 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5403ba90 con 0x7fad5809f870 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad4dffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad5ff1d640 10 monclient: _renew_subs 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad4dffb640 1 -- 172.21.15.110:0/2608809404 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5403be50 con 0x7fad5809f870 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad5ff1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad5ff1d640 1 -- 172.21.15.110:0/2608809404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5811d1b0 con 0x7fad5809f870 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad4dffb640 10 monclient: dump: 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:50.182 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.169+0000 7fad4ffff640 4 set_mon_vals no callback set 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.170+0000 7fad5ff1d640 10 monclient: _renew_subs 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.170+0000 7fad5ff1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.170+0000 7fad5ff1d640 1 -- 172.21.15.110:0/2608809404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad58134450 con 0x7fad5809f870 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.170+0000 7fad4dffb640 1 -- 172.21.15.110:0/2608809404 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fad54039040 con 0x7fad5809f870 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.170+0000 7fad5ff1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.170+0000 7fad5ff1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.170+0000 7fad5ff1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.170+0000 7fad5ff1d640 1 -- 172.21.15.110:0/2608809404 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad581346a0 con 0x7fad5809f870 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.171+0000 7fad4dffb640 1 --2- 172.21.15.110:0/2608809404 >> v2:172.21.15.177:6812/28201 conn(0x7fad3c038700 0x7fad3c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.171+0000 7fad4dffb640 1 -- 172.21.15.110:0/2608809404 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7fad5406b4e0 con 0x7fad5809f870 2024-09-15T03:58:50.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.171+0000 7fad5d4b8640 1 --2- 172.21.15.110:0/2608809404 >> v2:172.21.15.177:6812/28201 conn(0x7fad3c038700 0x7fad3c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.171+0000 7fad5d4b8640 10 monclient: get_auth_request con 0x7fad3c038700 auth_method 0 2024-09-15T03:58:50.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.174+0000 7fad5d4b8640 1 --2- 172.21.15.110:0/2608809404 >> v2:172.21.15.177:6812/28201 conn(0x7fad3c038700 0x7fad3c03abb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fad58134c90 tx=0x7fad48008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:50.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.175+0000 7fad4dffb640 1 -- 172.21.15.110:0/2608809404 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fad54042360 con 0x7fad5809f870 2024-09-15T03:58:50.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.175+0000 7fad4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:50.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.175+0000 7fad4dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:50.328 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.328+0000 7fad5ff1d640 1 -- 172.21.15.110:0/2608809404 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad58070950 con 0x7fad3c038700 2024-09-15T03:58:50.330 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.329+0000 7fad4dffb640 1 -- 172.21.15.110:0/2608809404 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+39252 (secure 0 0 0) 0x7fad58070950 con 0x7fad3c038700 2024-09-15T03:58:50.331 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:50.331 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-09-15T03:58:50.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.332+0000 7fad5ff1d640 1 -- 172.21.15.110:0/2608809404 >> v2:172.21.15.177:6812/28201 conn(0x7fad3c038700 msgr2=0x7fad3c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.332+0000 7fad5ff1d640 1 --2- 172.21.15.110:0/2608809404 >> v2:172.21.15.177:6812/28201 conn(0x7fad3c038700 0x7fad3c03abb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fad58134c90 tx=0x7fad48008040 comp rx=0 tx=0).stop 2024-09-15T03:58:50.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.332+0000 7fad5ff1d640 10 monclient: shutdown 2024-09-15T03:58:50.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.332+0000 7fad5ff1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:50.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.332+0000 7fad5ff1d640 1 -- 172.21.15.110:0/2608809404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad5809f870 msgr2=0x7fad58099660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.332+0000 7fad5ff1d640 1 --2- 172.21.15.110:0/2608809404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad5809f870 0x7fad58099660 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fad54035040 tx=0x7fad54032000 comp rx=0 tx=0).stop 2024-09-15T03:58:50.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.332+0000 7fad5ff1d640 1 -- 172.21.15.110:0/2608809404 shutdown_connections 2024-09-15T03:58:50.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.332+0000 7fad5ff1d640 1 --2- 172.21.15.110:0/2608809404 >> v2:172.21.15.177:6812/28201 conn(0x7fad3c038700 0x7fad3c03abb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.332+0000 7fad5ff1d640 1 --2- 172.21.15.110:0/2608809404 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad580a1a60 0x7fad5809a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.332+0000 7fad5ff1d640 1 --2- 172.21.15.110:0/2608809404 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad580a0380 0x7fad58099ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.332+0000 7fad5ff1d640 1 --2- 172.21.15.110:0/2608809404 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad5809f870 0x7fad58099660 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.332+0000 7fad5ff1d640 1 -- 172.21.15.110:0/2608809404 >> 172.21.15.110:0/2608809404 conn(0x7fad58063270 msgr2=0x7fad5809b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:50.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.332+0000 7fad5ff1d640 1 -- 172.21.15.110:0/2608809404 shutdown_connections 2024-09-15T03:58:50.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.332+0000 7fad5ff1d640 1 -- 172.21.15.110:0/2608809404 wait complete. 2024-09-15T03:58:50.343 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-15T03:58:48.363161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562077696,"kb_used":4777392,"kb_used_data":557300568,"kb_used_omap":443,"kb_used_meta":0,"kb_avail":557300304,"statfs":{"total":575567560704,"available":570675511296,"internally_reserved":270336,"allocated":570675781632,"data_stored":570675781632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":453772,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001072"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.072207+0000","last_change":"2024-09-15T03:58:44.072431+0000","last_active":"2024-09-15T03:58:44.072207+0000","last_peered":"2024-09-15T03:58:44.072207+0000","last_clean":"2024-09-15T03:58:44.072207+0000","last_became_active":"2024-09-15T03:58:42.157123+0000","last_became_peered":"2024-09-15T03:58:42.157123+0000","last_unstale":"2024-09-15T03:58:44.072207+0000","last_undegraded":"2024-09-15T03:58:44.072207+0000","last_fullsized":"2024-09-15T03:58:44.072207+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:46:37.797943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066518800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071760+0000","last_change":"2024-09-15T03:58:44.072012+0000","last_active":"2024-09-15T03:58:44.071760+0000","last_peered":"2024-09-15T03:58:44.071760+0000","last_clean":"2024-09-15T03:58:44.071760+0000","last_became_active":"2024-09-15T03:58:42.069324+0000","last_became_peered":"2024-09-15T03:58:42.069324+0000","last_unstale":"2024-09-15T03:58:44.071760+0000","last_undegraded":"2024-09-15T03:58:44.071760+0000","last_fullsized":"2024-09-15T03:58:44.071760+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:44:04.033148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040535300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071964+0000","last_change":"2024-09-15T03:58:44.072166+0000","last_active":"2024-09-15T03:58:44.071964+0000","last_peered":"2024-09-15T03:58:44.071964+0000","last_clean":"2024-09-15T03:58:44.071964+0000","last_became_active":"2024-09-15T03:58:42.155521+0000","last_became_peered":"2024-09-15T03:58:42.155521+0000","last_unstale":"2024-09-15T03:58:44.071964+0000","last_undegraded":"2024-09-15T03:58:44.071964+0000","last_fullsized":"2024-09-15T03:58:44.071964+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:42:01.920640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040527199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071921+0000","last_change":"2024-09-15T03:58:44.072137+0000","last_active":"2024-09-15T03:58:44.071921+0000","last_peered":"2024-09-15T03:58:44.071921+0000","last_clean":"2024-09-15T03:58:44.071921+0000","last_became_active":"2024-09-15T03:58:42.155317+0000","last_became_peered":"2024-09-15T03:58:42.155317+0000","last_unstale":"2024-09-15T03:58:44.071921+0000","last_undegraded":"2024-09-15T03:58:44.071921+0000","last_fullsized":"2024-09-15T03:58:44.071921+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:55:28.163584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041321199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.074749+0000","last_change":"2024-09-15T03:58:44.074749+0000","last_active":"2024-09-15T03:58:44.074749+0000","last_peered":"2024-09-15T03:58:44.074749+0000","last_clean":"2024-09-15T03:58:44.074749+0000","last_became_active":"2024-09-15T03:58:42.071376+0000","last_became_peered":"2024-09-15T03:58:42.071376+0000","last_unstale":"2024-09-15T03:58:44.074749+0000","last_undegraded":"2024-09-15T03:58:44.074749+0000","last_fullsized":"2024-09-15T03:58:44.074749+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:42:52.640034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080084100000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071218+0000","last_change":"2024-09-15T03:58:44.071453+0000","last_active":"2024-09-15T03:58:44.071218+0000","last_peered":"2024-09-15T03:58:44.071218+0000","last_clean":"2024-09-15T03:58:44.071218+0000","last_became_active":"2024-09-15T03:58:42.067012+0000","last_became_peered":"2024-09-15T03:58:42.067012+0000","last_unstale":"2024-09-15T03:58:44.071218+0000","last_undegraded":"2024-09-15T03:58:44.071218+0000","last_fullsized":"2024-09-15T03:58:44.071218+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:04:08.843151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040766700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071926+0000","last_change":"2024-09-15T03:58:44.072155+0000","last_active":"2024-09-15T03:58:44.071926+0000","last_peered":"2024-09-15T03:58:44.071926+0000","last_clean":"2024-09-15T03:58:44.071926+0000","last_became_active":"2024-09-15T03:58:42.069300+0000","last_became_peered":"2024-09-15T03:58:42.069300+0000","last_unstale":"2024-09-15T03:58:44.071926+0000","last_undegraded":"2024-09-15T03:58:44.071926+0000","last_fullsized":"2024-09-15T03:58:44.071926+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:28:34.591535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036612400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.074041+0000","last_change":"2024-09-15T03:58:44.074233+0000","last_active":"2024-09-15T03:58:44.074041+0000","last_peered":"2024-09-15T03:58:44.074041+0000","last_clean":"2024-09-15T03:58:44.074041+0000","last_became_active":"2024-09-15T03:58:42.071652+0000","last_became_peered":"2024-09-15T03:58:42.071652+0000","last_unstale":"2024-09-15T03:58:44.074041+0000","last_undegraded":"2024-09-15T03:58:44.074041+0000","last_fullsized":"2024-09-15T03:58:44.074041+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:31:44.288660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030212199999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":103,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T03:58:43.157400+0000","last_change":"2024-09-15T03:58:38.057397+0000","last_active":"2024-09-15T03:58:43.157400+0000","last_peered":"2024-09-15T03:58:43.157400+0000","last_clean":"2024-09-15T03:58:43.157400+0000","last_became_active":"2024-09-15T03:58:38.056673+0000","last_became_peered":"2024-09-15T03:58:38.056673+0000","last_unstale":"2024-09-15T03:58:43.157400+0000","last_undegraded":"2024-09-15T03:58:43.157400+0000","last_fullsized":"2024-09-15T03:58:43.157400+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_clean_scrub_stamp":"2024-09-15T03:58:37.036429+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:38:58.792683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795816,"kb_used_data":92883832,"kb_used_omap":56,"kb_used_meta":0,"kb_avail":92883800,"statfs":{"total":95927926784,"available":95113011200,"internally_reserved":32768,"allocated":95113043968,"data_stored":95113043968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58360,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795844,"kb_used_data":92883816,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92883772,"statfs":{"total":95927926784,"available":95112982528,"internally_reserved":45056,"allocated":95113027584,"data_stored":95113027584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48544,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797028,"kb_used_data":92882636,"kb_used_omap":109,"kb_used_meta":0,"kb_avail":92882588,"statfs":{"total":95927926784,"available":95111770112,"internally_reserved":49152,"allocated":95111819264,"data_stored":95111819264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112614,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795784,"kb_used_data":92883856,"kb_used_omap":55,"kb_used_meta":0,"kb_avail":92883832,"statfs":{"total":95927926784,"available":95113043968,"internally_reserved":24576,"allocated":95113068544,"data_stored":95113068544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":56457,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795840,"kb_used_data":92883828,"kb_used_omap":74,"kb_used_meta":0,"kb_avail":92883776,"statfs":{"total":95927926784,"available":95112986624,"internally_reserved":53248,"allocated":95113039872,"data_stored":95113039872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76777,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797080,"kb_used_data":92882600,"kb_used_omap":98,"kb_used_meta":0,"kb_avail":92882536,"statfs":{"total":95927926784,"available":95111716864,"internally_reserved":65536,"allocated":95111782400,"data_stored":95111782400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101020,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]}]}],"pool_statfs":[]}} 2024-09-15T03:58:50.344 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:58:50.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.435+0000 7fea8c99d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:50.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.435+0000 7fea8c99d640 10 monclient: build_initial_monmap 2024-09-15T03:58:50.437 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.435+0000 7fea8c99d640 10 monclient: monmap: 2024-09-15T03:58:50.437 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:50.437 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.437 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:50.436770+0000 2024-09-15T03:58:50.437 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:50.436770+0000 2024-09-15T03:58:50.437 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:50.437 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.437 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:50.437 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:50.437 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:50.437 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:50.437 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:50.437 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.436+0000 7fea8c99d640 1 Processor -- start 2024-09-15T03:58:50.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.436+0000 7fea8c99d640 1 -- start start 2024-09-15T03:58:50.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.436+0000 7fea8c99d640 10 monclient: init 2024-09-15T03:58:50.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.436+0000 7fea8c99d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:50.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.436+0000 7fea8c99d640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-15T03:58:50.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.436+0000 7fea8c99d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea880a1bf0 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:50.440 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.436+0000 7fea8c99d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea8809e780 0x7fea8809eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.436+0000 7fea8c99d640 10 monclient(hunting): picked mon.noname-b con 0x7fea8809e780 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:50.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.436+0000 7fea8c99d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fea88136a40 0x7fea88138e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.436+0000 7fea8c99d640 10 monclient(hunting): picked mon.noname-c con 0x7fea88136a40 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:50.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.436+0000 7fea8c99d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8809c9b0 con 0x7fea880a1bf0 2024-09-15T03:58:50.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea8c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:50.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea8c99d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea880a0c20 con 0x7fea88136a40 2024-09-15T03:58:50.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea8c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:50.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea8c99d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea8809cc90 con 0x7fea8809e780 2024-09-15T03:58:50.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea8c99d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:50.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea8c99d640 10 monclient(hunting): authenticate will time out at 754.081022s 2024-09-15T03:58:50.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea8809e780 0x7fea8809eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea8659c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fea880a1bf0 0x7fea88131810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:34146/0 (socket says 172.21.15.110:34146) 2024-09-15T03:58:50.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea8809e780 0x7fea8809eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43510/0 (socket says 172.21.15.110:43510) 2024-09-15T03:58:50.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea8659c640 1 -- 172.21.15.110:0/608079 learned_addr learned my addr 172.21.15.110:0/608079 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:50.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 10 monclient(hunting): get_auth_request con 0x7fea8809e780 auth_method 0 2024-09-15T03:58:50.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:50.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:50.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:50.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:50.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:50.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:50.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea86d9d640 1 --2- 172.21.15.110:0/608079 >> v2:172.21.15.177:3300/0 conn(0x7fea88136a40 0x7fea88138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea86d9d640 10 monclient(hunting): get_auth_request con 0x7fea88136a40 auth_method 0 2024-09-15T03:58:50.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea86d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:50.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea86d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:50.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea86d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:50.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 10 monclient(hunting): handle_auth_done global_id 4352 payload 274 2024-09-15T03:58:50.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 1 -- 172.21.15.110:0/608079 >> v2:172.21.15.177:3300/0 conn(0x7fea88136a40 msgr2=0x7fea88138e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:50.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 1 --2- 172.21.15.110:0/608079 >> v2:172.21.15.177:3300/0 conn(0x7fea88136a40 0x7fea88138e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 1 -- 172.21.15.110:0/608079 >> v1:172.21.15.110:6789/0 conn(0x7fea880a1bf0 legacy=0x7fea88131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:50.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:58:50.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 1 monclient: found mon.noname-b 2024-09-15T03:58:50.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:50.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:50.443 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.437+0000 7fea85d9b640 1 -- 172.21.15.110:0/608079 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea88131f20 con 0x7fea8809e780 2024-09-15T03:58:50.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea85d9b640 1 -- 172.21.15.110:0/608079 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea8809e780 msgr2=0x7fea8809eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:50.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea85d9b640 1 --2- 172.21.15.110:0/608079 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea8809e780 0x7fea8809eb50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fea8809d670 tx=0x7fea7c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9b9f6a0f8f90cf3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:50.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8559a640 1 -- 172.21.15.110:0/608079 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c02ae50 con 0x7fea8809e780 2024-09-15T03:58:50.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:50.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:50.449 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8559a640 10 monclient: dump: 2024-09-15T03:58:50.449 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:50.450 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.450 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.450 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.450 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:50.450 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.450 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:50.450 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:50.450 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:50.450 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8559a640 10 monclient: _finish_auth 0 2024-09-15T03:58:50.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:50.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8559a640 1 -- 172.21.15.110:0/608079 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c032440 con 0x7fea8809e780 2024-09-15T03:58:50.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8559a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:50.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8559a640 1 -- 172.21.15.110:0/608079 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c032820 con 0x7fea8809e780 2024-09-15T03:58:50.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:50.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:50.451 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8559a640 10 monclient: dump: 2024-09-15T03:58:50.451 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:50.451 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.451 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.451 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.451 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:50.451 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.451 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:50.451 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:50.451 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:50.451 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8c99d640 5 monclient: authenticate success, global_id 4352 2024-09-15T03:58:50.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8c99d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:50.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8c99d640 4 set_mon_vals no callback set 2024-09-15T03:58:50.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8c99d640 10 monclient: shutdown 2024-09-15T03:58:50.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8c99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:50.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8c99d640 1 -- 172.21.15.110:0/608079 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea8809e780 msgr2=0x7fea8809eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.440+0000 7fea8c99d640 1 --2- 172.21.15.110:0/608079 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea8809e780 0x7fea8809eb50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fea8809d670 tx=0x7fea7c0090f0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.441+0000 7fea8c99d640 1 -- 172.21.15.110:0/608079 shutdown_connections 2024-09-15T03:58:50.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.441+0000 7fea8c99d640 1 --2- 172.21.15.110:0/608079 >> v2:172.21.15.177:3300/0 conn(0x7fea88136a40 0x7fea88138e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.441+0000 7fea8c99d640 1 --2- 172.21.15.110:0/608079 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea8809e780 0x7fea8809eb50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.441+0000 7fea8c99d640 1 -- 172.21.15.110:0/608079 >> 172.21.15.110:0/608079 conn(0x7fea88062dd0 msgr2=0x7fea880631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:50.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.441+0000 7fea8c99d640 1 -- 172.21.15.110:0/608079 shutdown_connections 2024-09-15T03:58:50.452 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.441+0000 7fea8c99d640 1 -- 172.21.15.110:0/608079 wait complete. 2024-09-15T03:58:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.441+0000 7fea8c99d640 10 monclient: build_initial_monmap 2024-09-15T03:58:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.441+0000 7fea8c99d640 10 monclient: monmap: 2024-09-15T03:58:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:50.453 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.441+0000 7fea8c99d640 1 Processor -- start 2024-09-15T03:58:50.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.441+0000 7fea8c99d640 1 -- start start 2024-09-15T03:58:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.441+0000 7fea8c99d640 10 monclient: init 2024-09-15T03:58:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:58:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea8809e780 0x7fea88141720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 10 monclient(hunting): picked mon.noname-b con 0x7fea8809e780 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea880a1bf0 0x7fea88141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 10 monclient(hunting): picked mon.noname-a con 0x7fea880a1bf0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea88136a40 0x7fea88097690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 10 monclient(hunting): picked mon.noname-c con 0x7fea88136a40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea8809a010 con 0x7fea880a1bf0 2024-09-15T03:58:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:50.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea880a2e30 con 0x7fea8809e780 2024-09-15T03:58:50.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:50.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea88099710 con 0x7fea88136a40 2024-09-15T03:58:50.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:50.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8c99d640 10 monclient(hunting): authenticate will time out at 754.086259s 2024-09-15T03:58:50.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea880a1bf0 0x7fea88141c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea86d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea88136a40 0x7fea88097690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea880a1bf0 0x7fea88141c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53128/0 (socket says 172.21.15.110:53128) 2024-09-15T03:58:50.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea86d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea88136a40 0x7fea88097690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43522/0 (socket says 172.21.15.110:43522) 2024-09-15T03:58:50.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 1 -- 172.21.15.110:0/4140922145 learned_addr learned my addr 172.21.15.110:0/4140922145 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:50.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 10 monclient(hunting): get_auth_request con 0x7fea880a1bf0 auth_method 0 2024-09-15T03:58:50.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:50.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:50.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:50.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea86d9d640 10 monclient(hunting): get_auth_request con 0x7fea88136a40 auth_method 0 2024-09-15T03:58:50.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea86d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:50.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea86d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:50.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea86d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:50.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:50.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:50.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:50.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea86d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:50.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea86d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:50.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea86d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:50.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea8659c640 1 --2- 172.21.15.110:0/4140922145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea8809e780 0x7fea88141720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-09-15T03:58:50.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 1 -- 172.21.15.110:0/4140922145 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea88136a40 msgr2=0x7fea88097690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 1 --2- 172.21.15.110:0/4140922145 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea88136a40 0x7fea88097690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 1 -- 172.21.15.110:0/4140922145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea8809e780 msgr2=0x7fea88141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 1 --2- 172.21.15.110:0/4140922145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea8809e780 0x7fea88141720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:58:50.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 1 monclient: found mon.noname-a 2024-09-15T03:58:50.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:50.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:50.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea85d9b640 1 -- 172.21.15.110:0/4140922145 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea7c009390 con 0x7fea880a1bf0 2024-09-15T03:58:50.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.442+0000 7fea86d9d640 1 --2- 172.21.15.110:0/4140922145 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea88136a40 0x7fea88097690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:50.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.444+0000 7fea85d9b640 1 -- 172.21.15.110:0/4140922145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea880a1bf0 msgr2=0x7fea88141c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:50.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.444+0000 7fea85d9b640 1 --2- 172.21.15.110:0/4140922145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea880a1bf0 0x7fea88141c60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009880 tx=0x7fea7c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:50.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea767fc640 1 -- 172.21.15.110:0/4140922145 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c031c70 con 0x7fea880a1bf0 2024-09-15T03:58:50.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:50.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:50.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea767fc640 10 monclient: dump: 2024-09-15T03:58:50.458 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:50.458 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.458 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.458 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.458 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:50.458 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.458 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:50.458 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:50.458 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:50.458 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea767fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:50.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:50.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea767fc640 1 -- 172.21.15.110:0/4140922145 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c031dd0 con 0x7fea880a1bf0 2024-09-15T03:58:50.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea8c99d640 5 monclient: authenticate success, global_id 4326 2024-09-15T03:58:50.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea8c99d640 10 monclient: _renew_subs 2024-09-15T03:58:50.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea8c99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:50.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea8c99d640 1 -- 172.21.15.110:0/4140922145 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea88099200 con 0x7fea880a1bf0 2024-09-15T03:58:50.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea767fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:50.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea767fc640 1 -- 172.21.15.110:0/4140922145 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c03c630 con 0x7fea880a1bf0 2024-09-15T03:58:50.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:50.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:50.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea767fc640 10 monclient: dump: 2024-09-15T03:58:50.460 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:50.460 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.460 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.460 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.460 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:50.460 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.461 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:50.461 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:50.461 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:50.461 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea8c99d640 10 monclient: _renew_subs 2024-09-15T03:58:50.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea8c99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:50.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea8c99d640 1 -- 172.21.15.110:0/4140922145 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea8809b2f0 con 0x7fea880a1bf0 2024-09-15T03:58:50.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea84d99640 4 set_mon_vals no callback set 2024-09-15T03:58:50.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea767fc640 1 -- 172.21.15.110:0/4140922145 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fea7c03caa0 con 0x7fea880a1bf0 2024-09-15T03:58:50.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea8c99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:50.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea8c99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:50.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea8c99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:50.461 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.446+0000 7fea767fc640 1 --2- 172.21.15.110:0/4140922145 >> v2:172.21.15.177:6812/28201 conn(0x7fea68038700 0x7fea6803abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.445+0000 7fea8c99d640 1 -- 172.21.15.110:0/4140922145 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea88070950 con 0x7fea880a1bf0 2024-09-15T03:58:50.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.446+0000 7fea767fc640 1 -- 172.21.15.110:0/4140922145 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7fea7c06c310 con 0x7fea880a1bf0 2024-09-15T03:58:50.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.446+0000 7fea8659c640 1 --2- 172.21.15.110:0/4140922145 >> v2:172.21.15.177:6812/28201 conn(0x7fea68038700 0x7fea6803abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.446+0000 7fea8659c640 10 monclient: get_auth_request con 0x7fea68038700 auth_method 0 2024-09-15T03:58:50.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.448+0000 7fea8659c640 1 --2- 172.21.15.110:0/4140922145 >> v2:172.21.15.177:6812/28201 conn(0x7fea68038700 0x7fea6803abb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fea7800a840 tx=0x7fea78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:50.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.450+0000 7fea767fc640 1 -- 172.21.15.110:0/4140922145 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fea7c03b3a0 con 0x7fea880a1bf0 2024-09-15T03:58:50.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.450+0000 7fea767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:50.462 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.450+0000 7fea767fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:50.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.601+0000 7fea8c99d640 1 -- 172.21.15.110:0/4140922145 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea88099df0 con 0x7fea68038700 2024-09-15T03:58:50.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.603+0000 7fea767fc640 1 -- 172.21.15.110:0/4140922145 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+39252 (secure 0 0 0) 0x7fea88099df0 con 0x7fea68038700 2024-09-15T03:58:50.605 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T03:58:50.605 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-09-15T03:58:50.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.605+0000 7fea8c99d640 1 -- 172.21.15.110:0/4140922145 >> v2:172.21.15.177:6812/28201 conn(0x7fea68038700 msgr2=0x7fea6803abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.605+0000 7fea8c99d640 1 --2- 172.21.15.110:0/4140922145 >> v2:172.21.15.177:6812/28201 conn(0x7fea68038700 0x7fea6803abb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fea7800a840 tx=0x7fea78008040 comp rx=0 tx=0).stop 2024-09-15T03:58:50.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.605+0000 7fea8c99d640 10 monclient: shutdown 2024-09-15T03:58:50.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.605+0000 7fea8c99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:50.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.605+0000 7fea8c99d640 1 -- 172.21.15.110:0/4140922145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea880a1bf0 msgr2=0x7fea88141c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.605+0000 7fea8c99d640 1 --2- 172.21.15.110:0/4140922145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea880a1bf0 0x7fea88141c60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009880 tx=0x7fea7c032000 comp rx=0 tx=0).stop 2024-09-15T03:58:50.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.605+0000 7fea8c99d640 1 -- 172.21.15.110:0/4140922145 shutdown_connections 2024-09-15T03:58:50.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.605+0000 7fea8c99d640 1 --2- 172.21.15.110:0/4140922145 >> v2:172.21.15.177:6812/28201 conn(0x7fea68038700 0x7fea6803abb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.605+0000 7fea8c99d640 1 --2- 172.21.15.110:0/4140922145 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fea88136a40 0x7fea88097690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.605+0000 7fea8c99d640 1 --2- 172.21.15.110:0/4140922145 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fea880a1bf0 0x7fea88141c60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.607 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.605+0000 7fea8c99d640 1 --2- 172.21.15.110:0/4140922145 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fea8809e780 0x7fea88141720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.605+0000 7fea8c99d640 1 -- 172.21.15.110:0/4140922145 >> 172.21.15.110:0/4140922145 conn(0x7fea88062dd0 msgr2=0x7fea880a4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:50.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.605+0000 7fea8c99d640 1 -- 172.21.15.110:0/4140922145 shutdown_connections 2024-09-15T03:58:50.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.605+0000 7fea8c99d640 1 -- 172.21.15.110:0/4140922145 wait complete. 2024-09-15T03:58:50.616 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-15T03:58:50.363510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562077696,"kb_used":4777392,"kb_used_data":557300568,"kb_used_omap":443,"kb_used_meta":0,"kb_avail":557300304,"statfs":{"total":575567560704,"available":570675511296,"internally_reserved":270336,"allocated":570675781632,"data_stored":570675781632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":453772,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.294283"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.072207+0000","last_change":"2024-09-15T03:58:44.072431+0000","last_active":"2024-09-15T03:58:44.072207+0000","last_peered":"2024-09-15T03:58:44.072207+0000","last_clean":"2024-09-15T03:58:44.072207+0000","last_became_active":"2024-09-15T03:58:42.157123+0000","last_became_peered":"2024-09-15T03:58:42.157123+0000","last_unstale":"2024-09-15T03:58:44.072207+0000","last_undegraded":"2024-09-15T03:58:44.072207+0000","last_fullsized":"2024-09-15T03:58:44.072207+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:46:37.797943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066518800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071760+0000","last_change":"2024-09-15T03:58:44.072012+0000","last_active":"2024-09-15T03:58:44.071760+0000","last_peered":"2024-09-15T03:58:44.071760+0000","last_clean":"2024-09-15T03:58:44.071760+0000","last_became_active":"2024-09-15T03:58:42.069324+0000","last_became_peered":"2024-09-15T03:58:42.069324+0000","last_unstale":"2024-09-15T03:58:44.071760+0000","last_undegraded":"2024-09-15T03:58:44.071760+0000","last_fullsized":"2024-09-15T03:58:44.071760+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:44:04.033148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040535300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071964+0000","last_change":"2024-09-15T03:58:44.072166+0000","last_active":"2024-09-15T03:58:44.071964+0000","last_peered":"2024-09-15T03:58:44.071964+0000","last_clean":"2024-09-15T03:58:44.071964+0000","last_became_active":"2024-09-15T03:58:42.155521+0000","last_became_peered":"2024-09-15T03:58:42.155521+0000","last_unstale":"2024-09-15T03:58:44.071964+0000","last_undegraded":"2024-09-15T03:58:44.071964+0000","last_fullsized":"2024-09-15T03:58:44.071964+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:42:01.920640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040527199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071921+0000","last_change":"2024-09-15T03:58:44.072137+0000","last_active":"2024-09-15T03:58:44.071921+0000","last_peered":"2024-09-15T03:58:44.071921+0000","last_clean":"2024-09-15T03:58:44.071921+0000","last_became_active":"2024-09-15T03:58:42.155317+0000","last_became_peered":"2024-09-15T03:58:42.155317+0000","last_unstale":"2024-09-15T03:58:44.071921+0000","last_undegraded":"2024-09-15T03:58:44.071921+0000","last_fullsized":"2024-09-15T03:58:44.071921+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:55:28.163584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041321199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.074749+0000","last_change":"2024-09-15T03:58:44.074749+0000","last_active":"2024-09-15T03:58:44.074749+0000","last_peered":"2024-09-15T03:58:44.074749+0000","last_clean":"2024-09-15T03:58:44.074749+0000","last_became_active":"2024-09-15T03:58:42.071376+0000","last_became_peered":"2024-09-15T03:58:42.071376+0000","last_unstale":"2024-09-15T03:58:44.074749+0000","last_undegraded":"2024-09-15T03:58:44.074749+0000","last_fullsized":"2024-09-15T03:58:44.074749+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:42:52.640034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080084100000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071218+0000","last_change":"2024-09-15T03:58:44.071453+0000","last_active":"2024-09-15T03:58:44.071218+0000","last_peered":"2024-09-15T03:58:44.071218+0000","last_clean":"2024-09-15T03:58:44.071218+0000","last_became_active":"2024-09-15T03:58:42.067012+0000","last_became_peered":"2024-09-15T03:58:42.067012+0000","last_unstale":"2024-09-15T03:58:44.071218+0000","last_undegraded":"2024-09-15T03:58:44.071218+0000","last_fullsized":"2024-09-15T03:58:44.071218+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:04:08.843151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040766700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.071926+0000","last_change":"2024-09-15T03:58:44.072155+0000","last_active":"2024-09-15T03:58:44.071926+0000","last_peered":"2024-09-15T03:58:44.071926+0000","last_clean":"2024-09-15T03:58:44.071926+0000","last_became_active":"2024-09-15T03:58:42.069300+0000","last_became_peered":"2024-09-15T03:58:42.069300+0000","last_unstale":"2024-09-15T03:58:44.071926+0000","last_undegraded":"2024-09-15T03:58:44.071926+0000","last_fullsized":"2024-09-15T03:58:44.071926+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:28:34.591535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036612400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T03:58:44.074041+0000","last_change":"2024-09-15T03:58:44.074233+0000","last_active":"2024-09-15T03:58:44.074041+0000","last_peered":"2024-09-15T03:58:44.074041+0000","last_clean":"2024-09-15T03:58:44.074041+0000","last_became_active":"2024-09-15T03:58:42.071652+0000","last_became_peered":"2024-09-15T03:58:42.071652+0000","last_unstale":"2024-09-15T03:58:44.074041+0000","last_undegraded":"2024-09-15T03:58:44.074041+0000","last_fullsized":"2024-09-15T03:58:44.074041+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:31:44.288660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030212199999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":103,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-15T03:58:43.157400+0000","last_change":"2024-09-15T03:58:38.057397+0000","last_active":"2024-09-15T03:58:43.157400+0000","last_peered":"2024-09-15T03:58:43.157400+0000","last_clean":"2024-09-15T03:58:43.157400+0000","last_became_active":"2024-09-15T03:58:38.056673+0000","last_became_peered":"2024-09-15T03:58:38.056673+0000","last_unstale":"2024-09-15T03:58:43.157400+0000","last_undegraded":"2024-09-15T03:58:43.157400+0000","last_fullsized":"2024-09-15T03:58:43.157400+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_clean_scrub_stamp":"2024-09-15T03:58:37.036429+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:38:58.792683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795816,"kb_used_data":92883832,"kb_used_omap":56,"kb_used_meta":0,"kb_avail":92883800,"statfs":{"total":95927926784,"available":95113011200,"internally_reserved":32768,"allocated":95113043968,"data_stored":95113043968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58360,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795844,"kb_used_data":92883816,"kb_used_omap":47,"kb_used_meta":0,"kb_avail":92883772,"statfs":{"total":95927926784,"available":95112982528,"internally_reserved":45056,"allocated":95113027584,"data_stored":95113027584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48544,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797028,"kb_used_data":92882636,"kb_used_omap":109,"kb_used_meta":0,"kb_avail":92882588,"statfs":{"total":95927926784,"available":95111770112,"internally_reserved":49152,"allocated":95111819264,"data_stored":95111819264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":112614,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795784,"kb_used_data":92883856,"kb_used_omap":55,"kb_used_meta":0,"kb_avail":92883832,"statfs":{"total":95927926784,"available":95113043968,"internally_reserved":24576,"allocated":95113068544,"data_stored":95113068544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":56457,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":795840,"kb_used_data":92883828,"kb_used_omap":74,"kb_used_meta":0,"kb_avail":92883776,"statfs":{"total":95927926784,"available":95112986624,"internally_reserved":53248,"allocated":95113039872,"data_stored":95113039872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76777,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":797080,"kb_used_data":92882600,"kb_used_omap":98,"kb_used_meta":0,"kb_avail":92882536,"statfs":{"total":95927926784,"available":95111716864,"internally_reserved":65536,"allocated":95111782400,"data_stored":95111782400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":101020,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]}]}],"pool_statfs":[]}} 2024-09-15T03:58:50.617 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-15T03:58:50.618 DEBUG:tasks.mon_thrash:Yielding 2024-09-15T03:58:50.618 INFO:teuthology.run_tasks:Running task exec... 2024-09-15T03:58:50.627 INFO:teuthology.task.exec:Executing custom commands... 2024-09-15T03:58:50.628 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi177.front.sepia.ceph.com 2024-09-15T03:58:50.628 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T03:58:50.628 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-15T03:58:50.628 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726372730, revive delay: 90.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-15T03:58:50.628 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:58:50.629 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:58:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.721+0000 7f47506c0640 10 monclient: get_monmap_and_config 2024-09-15T03:58:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.721+0000 7f47506c0640 10 monclient: build_initial_monmap 2024-09-15T03:58:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.721+0000 7f47506c0640 10 monclient: monmap: 2024-09-15T03:58:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:50.722152+0000 2024-09-15T03:58:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:50.722152+0000 2024-09-15T03:58:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.722 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:50.723 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:50.723 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:50.723 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:50.723 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:50.723 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.721+0000 7f47506c0640 1 Processor -- start 2024-09-15T03:58:50.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.721+0000 7f47506c0640 1 -- start start 2024-09-15T03:58:50.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.721+0000 7f47506c0640 10 monclient: init 2024-09-15T03:58:50.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f47506c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:50.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f47506c0640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-15T03:58:50.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f47506c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47480a1a60 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:50.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f47506c0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131640 0x7f4748131a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f47506c0640 10 monclient(hunting): picked mon.noname-b con 0x7f4748131640 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:50.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f47506c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4748131f50 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:50.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f47506c0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47480a1260 con 0x7f4748131f50 2024-09-15T03:58:50.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f47506c0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474809c870 con 0x7f47480a1a60 2024-09-15T03:58:50.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f47506c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:50.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f47506c0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47480a0c60 con 0x7f4748131640 2024-09-15T03:58:50.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f47506c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:50.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f47506c0640 10 monclient(hunting): authenticate will time out at 754.366395s 2024-09-15T03:58:50.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131640 0x7f4748131a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131640 0x7f4748131a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43536/0 (socket says 172.21.15.110:43536) 2024-09-15T03:58:50.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474ec5d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4748131f50 0x7f4748138b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:34152/0 (socket says 172.21.15.110:34152) 2024-09-15T03:58:50.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 1 -- 172.21.15.110:0/3520265689 learned_addr learned my addr 172.21.15.110:0/3520265689 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:50.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 10 monclient(hunting): get_auth_request con 0x7f4748131640 auth_method 0 2024-09-15T03:58:50.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:50.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:50.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:50.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:50.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:50.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:50.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 10 monclient(hunting): handle_auth_done global_id 4367 payload 274 2024-09-15T03:58:50.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 1 -- 172.21.15.110:0/3520265689 >> v1:172.21.15.177:6789/0 conn(0x7f47480a1a60 legacy=0x7f474809d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:50.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 1 -- 172.21.15.110:0/3520265689 >> v1:172.21.15.110:6789/0 conn(0x7f4748131f50 legacy=0x7f4748138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:50.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 10 monclient: _finish_hunting 0 2024-09-15T03:58:50.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 1 monclient: found mon.noname-b 2024-09-15T03:58:50.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:50.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:50.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.722+0000 7f474dc5b640 1 -- 172.21.15.110:0/3520265689 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474809cb10 con 0x7f4748131640 2024-09-15T03:58:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f474dc5b640 1 -- 172.21.15.110:0/3520265689 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131640 msgr2=0x7f4748131a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f474dc5b640 1 --2- 172.21.15.110:0/3520265689 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131640 0x7f4748131a10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f473c009940 tx=0x7f473c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=825a8e1a04e6b2c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f474d45a640 1 -- 172.21.15.110:0/3520265689 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c02ae30 con 0x7f4748131640 2024-09-15T03:58:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f474d45a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:50.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f474d45a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:50.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f474d45a640 10 monclient: dump: 2024-09-15T03:58:50.736 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:50.736 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.736 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.736 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.736 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:50.736 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.736 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:50.736 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:50.736 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:50.736 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f474d45a640 10 monclient: _finish_auth 0 2024-09-15T03:58:50.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f474d45a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:50.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f474d45a640 1 -- 172.21.15.110:0/3520265689 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c02f4b0 con 0x7f4748131640 2024-09-15T03:58:50.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f47506c0640 5 monclient: authenticate success, global_id 4367 2024-09-15T03:58:50.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f47506c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:50.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f474d45a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:50.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f47506c0640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:50.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f474d45a640 1 -- 172.21.15.110:0/3520265689 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c02f850 con 0x7f4748131640 2024-09-15T03:58:50.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f47506c0640 4 set_mon_vals no callback set 2024-09-15T03:58:50.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f474d45a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:50.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f474d45a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:50.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.725+0000 7f474d45a640 10 monclient: dump: 2024-09-15T03:58:50.738 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:50.738 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.738 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.738 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.738 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:50.738 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.738 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:50.738 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:50.738 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:50.738 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.726+0000 7f47506c0640 10 monclient: shutdown 2024-09-15T03:58:50.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.726+0000 7f47506c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:50.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.726+0000 7f47506c0640 1 -- 172.21.15.110:0/3520265689 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131640 msgr2=0x7f4748131a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.726+0000 7f47506c0640 1 --2- 172.21.15.110:0/3520265689 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131640 0x7f4748131a10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f473c009940 tx=0x7f473c0091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.726+0000 7f47506c0640 1 -- 172.21.15.110:0/3520265689 shutdown_connections 2024-09-15T03:58:50.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.726+0000 7f47506c0640 1 --2- 172.21.15.110:0/3520265689 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131640 0x7f4748131a10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.726+0000 7f47506c0640 1 -- 172.21.15.110:0/3520265689 >> 172.21.15.110:0/3520265689 conn(0x7f4748063270 msgr2=0x7f4748063670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:50.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.726+0000 7f47506c0640 1 -- 172.21.15.110:0/3520265689 shutdown_connections 2024-09-15T03:58:50.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.726+0000 7f47506c0640 1 -- 172.21.15.110:0/3520265689 wait complete. 2024-09-15T03:58:50.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.726+0000 7f47506c0640 10 monclient: build_initial_monmap 2024-09-15T03:58:50.739 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.726+0000 7f47506c0640 10 monclient: monmap: 2024-09-15T03:58:50.739 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:50.739 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:50.739 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:50.739 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:50.739 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:50.739 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:50.740 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 1 Processor -- start 2024-09-15T03:58:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 1 -- start start 2024-09-15T03:58:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 10 monclient: init 2024-09-15T03:58:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:50.740 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:58:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47480a1a60 0x7f4748130de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 10 monclient(hunting): picked mon.noname-b con 0x7f47480a1a60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4748131640 0x7f4748095a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 10 monclient(hunting): picked mon.noname-a con 0x7f4748131640 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131f50 0x7f4748095fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 10 monclient(hunting): picked mon.noname-c con 0x7f4748131f50 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:50.741 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4748098a50 con 0x7f4748131640 2024-09-15T03:58:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47480a1450 con 0x7f47480a1a60 2024-09-15T03:58:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4748097420 con 0x7f4748131f50 2024-09-15T03:58:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f47506c0640 10 monclient(hunting): authenticate will time out at 754.371651s 2024-09-15T03:58:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474dc5b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4748131640 0x7f4748095a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474ec5d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131f50 0x7f4748095fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474dc5b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4748131640 0x7f4748095a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53134/0 (socket says 172.21.15.110:53134) 2024-09-15T03:58:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474dc5b640 1 -- 172.21.15.110:0/1798103297 learned_addr learned my addr 172.21.15.110:0/1798103297 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:50.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474dc5b640 10 monclient(hunting): get_auth_request con 0x7f4748131640 auth_method 0 2024-09-15T03:58:50.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474dc5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:50.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474ec5d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131f50 0x7f4748095fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43548/0 (socket says 172.21.15.110:43548) 2024-09-15T03:58:50.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474dc5b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:50.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474dc5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:50.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474ec5d640 10 monclient(hunting): get_auth_request con 0x7f4748131f50 auth_method 0 2024-09-15T03:58:50.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474ec5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:50.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474ec5d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:50.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474ec5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:50.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474e45c640 1 --2- 172.21.15.110:0/1798103297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47480a1a60 0x7f4748130de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474e45c640 10 monclient(hunting): get_auth_request con 0x7f47480a1a60 auth_method 0 2024-09-15T03:58:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474e45c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474e45c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474e45c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474dc5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474dc5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474dc5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474ec5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474ec5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474ec5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474e45c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474e45c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.727+0000 7f474e45c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:50.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.728+0000 7f474ec5d640 10 monclient(hunting): handle_auth_done global_id 4370 payload 824 2024-09-15T03:58:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.728+0000 7f474ec5d640 1 -- 172.21.15.110:0/1798103297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47480a1a60 msgr2=0x7f4748130de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.728+0000 7f474ec5d640 1 --2- 172.21.15.110:0/1798103297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47480a1a60 0x7f4748130de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.728+0000 7f474ec5d640 1 -- 172.21.15.110:0/1798103297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4748131640 msgr2=0x7f4748095a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.728+0000 7f474ec5d640 1 --2- 172.21.15.110:0/1798103297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4748131640 0x7f4748095a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.728+0000 7f474ec5d640 10 monclient: _finish_hunting 0 2024-09-15T03:58:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.728+0000 7f474ec5d640 1 monclient: found mon.noname-c 2024-09-15T03:58:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.728+0000 7f474ec5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.728+0000 7f474ec5d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.728+0000 7f474ec5d640 1 -- 172.21.15.110:0/1798103297 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c009450 con 0x7f4748131f50 2024-09-15T03:58:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.728+0000 7f474dc5b640 1 --2- 172.21.15.110:0/1798103297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4748131640 0x7f4748095a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:50.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f474ec5d640 1 -- 172.21.15.110:0/1798103297 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131f50 msgr2=0x7f4748095fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f474ec5d640 1 --2- 172.21.15.110:0/1798103297 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131f50 0x7f4748095fb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f473800c790 tx=0x7f4738004e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47467fc640 1 -- 172.21.15.110:0/1798103297 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738005d60 con 0x7f4748131f50 2024-09-15T03:58:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47467fc640 10 monclient: dump: 2024-09-15T03:58:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:50.746 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:50.747 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47467fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47467fc640 1 -- 172.21.15.110:0/1798103297 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473800a640 con 0x7f4748131f50 2024-09-15T03:58:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47506c0640 5 monclient: authenticate success, global_id 4370 2024-09-15T03:58:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47467fc640 1 -- 172.21.15.110:0/1798103297 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473800aa00 con 0x7f4748131f50 2024-09-15T03:58:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47506c0640 10 monclient: _renew_subs 2024-09-15T03:58:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47506c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:50.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47506c0640 1 -- 172.21.15.110:0/1798103297 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4748098770 con 0x7f4748131f50 2024-09-15T03:58:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47467fc640 10 monclient: dump: 2024-09-15T03:58:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:50.748 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:50.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f474cc59640 4 set_mon_vals no callback set 2024-09-15T03:58:50.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47506c0640 10 monclient: _renew_subs 2024-09-15T03:58:50.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47506c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:50.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.730+0000 7f47506c0640 1 -- 172.21.15.110:0/1798103297 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4748097a20 con 0x7f4748131f50 2024-09-15T03:58:50.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.731+0000 7f47467fc640 1 -- 172.21.15.110:0/1798103297 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f47380070f0 con 0x7f4748131f50 2024-09-15T03:58:50.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.731+0000 7f47506c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:50.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.731+0000 7f47506c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:50.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.731+0000 7f47506c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:50.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.731+0000 7f47506c0640 1 -- 172.21.15.110:0/1798103297 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4748098c90 con 0x7f4748131f50 2024-09-15T03:58:50.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.731+0000 7f47467fc640 1 --2- 172.21.15.110:0/1798103297 >> v2:172.21.15.177:6812/28201 conn(0x7f4734038700 0x7f473403abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:50.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.731+0000 7f47467fc640 1 -- 172.21.15.110:0/1798103297 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f473804b320 con 0x7f4748131f50 2024-09-15T03:58:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.731+0000 7f474e45c640 1 --2- 172.21.15.110:0/1798103297 >> v2:172.21.15.177:6812/28201 conn(0x7f4734038700 0x7f473403abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.731+0000 7f474e45c640 10 monclient: get_auth_request con 0x7f4734038700 auth_method 0 2024-09-15T03:58:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.734+0000 7f474e45c640 1 --2- 172.21.15.110:0/1798103297 >> v2:172.21.15.177:6812/28201 conn(0x7f4734038700 0x7f473403abb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4740007fd0 tx=0x7f4740007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.735+0000 7f47467fc640 1 -- 172.21.15.110:0/1798103297 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f47380073a0 con 0x7f4748131f50 2024-09-15T03:58:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.735+0000 7f47467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:50.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.735+0000 7f47467fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:50.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.947+0000 7f47506c0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:58:50.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.947+0000 7f47506c0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:58:50.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.947+0000 7f47506c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:50.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.947+0000 7f47506c0640 1 -- 172.21.15.110:0/1798103297 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4748070950 con 0x7f4748131f50 2024-09-15T03:58:50.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.947+0000 7f47467fc640 1 -- 172.21.15.110:0/1798103297 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f4738010af0 con 0x7f4748131f50 2024-09-15T03:58:50.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.947+0000 7f47467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:58:50.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.947+0000 7f47467fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:50.950 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:58:50.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.949+0000 7f47506c0640 1 -- 172.21.15.110:0/1798103297 >> v2:172.21.15.177:6812/28201 conn(0x7f4734038700 msgr2=0x7f473403abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.949+0000 7f47506c0640 1 --2- 172.21.15.110:0/1798103297 >> v2:172.21.15.177:6812/28201 conn(0x7f4734038700 0x7f473403abb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4740007fd0 tx=0x7f4740007e10 comp rx=0 tx=0).stop 2024-09-15T03:58:50.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.949+0000 7f47506c0640 10 monclient: shutdown 2024-09-15T03:58:50.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.949+0000 7f47506c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:50.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.949+0000 7f47506c0640 1 -- 172.21.15.110:0/1798103297 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131f50 msgr2=0x7f4748095fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:50.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.949+0000 7f47506c0640 1 --2- 172.21.15.110:0/1798103297 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131f50 0x7f4748095fb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f473800c790 tx=0x7f4738004e20 comp rx=0 tx=0).stop 2024-09-15T03:58:50.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.949+0000 7f47506c0640 1 -- 172.21.15.110:0/1798103297 shutdown_connections 2024-09-15T03:58:50.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.949+0000 7f47506c0640 1 --2- 172.21.15.110:0/1798103297 >> v2:172.21.15.177:6812/28201 conn(0x7f4734038700 0x7f473403abb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.949+0000 7f47506c0640 1 --2- 172.21.15.110:0/1798103297 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4748131f50 0x7f4748095fb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.949+0000 7f47506c0640 1 --2- 172.21.15.110:0/1798103297 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4748131640 0x7f4748095a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.949+0000 7f47506c0640 1 --2- 172.21.15.110:0/1798103297 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f47480a1a60 0x7f4748130de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:50.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.949+0000 7f47506c0640 1 -- 172.21.15.110:0/1798103297 >> 172.21.15.110:0/1798103297 conn(0x7f4748063270 msgr2=0x7f474809dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:50.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.949+0000 7f47506c0640 1 -- 172.21.15.110:0/1798103297 shutdown_connections 2024-09-15T03:58:50.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:50.949+0000 7f47506c0640 1 -- 172.21.15.110:0/1798103297 wait complete. 2024-09-15T03:58:50.960 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:58:50.961 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:58:50.961 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:58:50.961 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:58:51.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.053+0000 7f4ed06d4640 10 monclient: get_monmap_and_config 2024-09-15T03:58:51.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.053+0000 7f4ed06d4640 10 monclient: build_initial_monmap 2024-09-15T03:58:51.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.053+0000 7f4ed06d4640 10 monclient: monmap: 2024-09-15T03:58:51.054 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:51.054 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.054 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:51.054498+0000 2024-09-15T03:58:51.055 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:51.054498+0000 2024-09-15T03:58:51.055 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:51.055 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.055 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:51.055 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:51.055 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:51.055 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:51.055 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:51.055 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 1 Processor -- start 2024-09-15T03:58:51.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 1 -- start start 2024-09-15T03:58:51.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 10 monclient: init 2024-09-15T03:58:51.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:51.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-15T03:58:51.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4ec80a3410 0x7f4ec806e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec80a3410 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:51.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ec8069ff0 0x7f4ec806ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec8069ff0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:51.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ec809d4f0 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:51.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec80a37e0 con 0x7f4ec809d4f0 2024-09-15T03:58:51.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec8069330 con 0x7f4ec80a3410 2024-09-15T03:58:51.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec8131fa0 con 0x7f4ec8069ff0 2024-09-15T03:58:51.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:51.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ed06d4640 10 monclient(hunting): authenticate will time out at 754.698744s 2024-09-15T03:58:51.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ecdc6f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ec8069ff0 0x7f4ec806ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ecec71640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4ec809d4f0 0x7f4ec809d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:34160/0 (socket says 172.21.15.110:34160) 2024-09-15T03:58:51.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ecec71640 1 -- 172.21.15.110:0/3164993556 learned_addr learned my addr 172.21.15.110:0/3164993556 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:51.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ecdc6f640 1 --2- 172.21.15.110:0/3164993556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ec8069ff0 0x7f4ec806ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43562/0 (socket says 172.21.15.110:43562) 2024-09-15T03:58:51.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ecdc6f640 10 monclient(hunting): get_auth_request con 0x7f4ec8069ff0 auth_method 0 2024-09-15T03:58:51.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ecdc6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ecdc6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ecdc6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ecdc6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ecdc6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.054+0000 7f4ecdc6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.055+0000 7f4ecdc6f640 10 monclient(hunting): handle_auth_done global_id 4373 payload 274 2024-09-15T03:58:51.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.055+0000 7f4ecdc6f640 1 -- 172.21.15.110:0/3164993556 >> v2:172.21.15.177:3300/0 conn(0x7f4ec80a3410 msgr2=0x7f4ec806e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:51.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.055+0000 7f4ece470640 1 --2- 172.21.15.110:0/3164993556 >> v2:172.21.15.177:3300/0 conn(0x7f4ec80a3410 0x7f4ec806e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.055+0000 7f4ecdc6f640 1 --2- 172.21.15.110:0/3164993556 >> v2:172.21.15.177:3300/0 conn(0x7f4ec80a3410 0x7f4ec806e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.055+0000 7f4ecdc6f640 1 -- 172.21.15.110:0/3164993556 >> v1:172.21.15.110:6789/0 conn(0x7f4ec809d4f0 legacy=0x7f4ec809d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:51.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.055+0000 7f4ecdc6f640 10 monclient: _finish_hunting 0 2024-09-15T03:58:51.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.055+0000 7f4ecdc6f640 1 monclient: found mon.noname-b 2024-09-15T03:58:51.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.055+0000 7f4ecdc6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:51.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.055+0000 7f4ecdc6f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:51.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.055+0000 7f4ecdc6f640 1 -- 172.21.15.110:0/3164993556 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec809f000 con 0x7f4ec8069ff0 2024-09-15T03:58:51.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.055+0000 7f4ece470640 10 monclient: get_auth_request con 0x7f4ec80a3410 auth_method 0 2024-09-15T03:58:51.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.055+0000 7f4ece470640 1 --2- 172.21.15.110:0/3164993556 >> v2:172.21.15.177:3300/0 conn(0x7f4ec80a3410 0x7f4ec806e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T03:58:51.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.057+0000 7f4ecdc6f640 1 -- 172.21.15.110:0/3164993556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ec8069ff0 msgr2=0x7f4ec806ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:51.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.057+0000 7f4ecdc6f640 1 --2- 172.21.15.110:0/3164993556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ec8069ff0 0x7f4ec806ea00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc009e60 tx=0x7f4ebc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87dffa6663e32c0c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ecd46e640 1 -- 172.21.15.110:0/3164993556 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc007840 con 0x7f4ec8069ff0 2024-09-15T03:58:51.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ecd46e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ecd46e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:51.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ecd46e640 10 monclient: dump: 2024-09-15T03:58:51.069 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.069 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.069 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.070 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.070 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.070 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.070 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.070 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.070 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.070 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ecd46e640 10 monclient: _finish_auth 0 2024-09-15T03:58:51.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ecd46e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:51.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ecd46e640 1 -- 172.21.15.110:0/3164993556 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc02d050 con 0x7f4ec8069ff0 2024-09-15T03:58:51.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ed06d4640 5 monclient: authenticate success, global_id 4373 2024-09-15T03:58:51.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ed06d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:51.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ecd46e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ed06d4640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ecd46e640 1 -- 172.21.15.110:0/3164993556 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc007bd0 con 0x7f4ec8069ff0 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ed06d4640 4 set_mon_vals no callback set 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ecd46e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ecd46e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ecd46e640 10 monclient: dump: 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.071 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.072 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ed06d4640 10 monclient: shutdown 2024-09-15T03:58:51.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ed06d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:51.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ed06d4640 1 -- 172.21.15.110:0/3164993556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ec8069ff0 msgr2=0x7f4ec806ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ed06d4640 1 --2- 172.21.15.110:0/3164993556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ec8069ff0 0x7f4ec806ea00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc009e60 tx=0x7f4ebc009780 comp rx=0 tx=0).stop 2024-09-15T03:58:51.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ed06d4640 1 -- 172.21.15.110:0/3164993556 shutdown_connections 2024-09-15T03:58:51.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ed06d4640 1 --2- 172.21.15.110:0/3164993556 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ec8069ff0 0x7f4ec806ea00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ed06d4640 1 --2- 172.21.15.110:0/3164993556 >> v2:172.21.15.177:3300/0 conn(0x7f4ec80a3410 0x7f4ec806e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ed06d4640 1 -- 172.21.15.110:0/3164993556 >> 172.21.15.110:0/3164993556 conn(0x7f4ec80635b0 msgr2=0x7f4ec80a2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:51.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ed06d4640 1 -- 172.21.15.110:0/3164993556 shutdown_connections 2024-09-15T03:58:51.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.058+0000 7f4ed06d4640 1 -- 172.21.15.110:0/3164993556 wait complete. 2024-09-15T03:58:51.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 10 monclient: build_initial_monmap 2024-09-15T03:58:51.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 10 monclient: monmap: 2024-09-15T03:58:51.073 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:51.073 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:51.073 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:51.073 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:51.073 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:51.073 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.073 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:51.073 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:51.073 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:51.073 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 1 Processor -- start 2024-09-15T03:58:51.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 1 -- start start 2024-09-15T03:58:51.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 10 monclient: init 2024-09-15T03:58:51.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:51.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:58:51.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ec8069ff0 0x7f4ec809bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 10 monclient(hunting): picked mon.noname-b con 0x7f4ec8069ff0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:51.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec809d4f0 0x7f4ec812e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 10 monclient(hunting): picked mon.noname-a con 0x7f4ec809d4f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:51.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ec80a3410 0x7f4ec8119160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 10 monclient(hunting): picked mon.noname-c con 0x7f4ec80a3410 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:51.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec8068710 con 0x7f4ec809d4f0 2024-09-15T03:58:51.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec80a14f0 con 0x7f4ec8069ff0 2024-09-15T03:58:51.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec809ca90 con 0x7f4ec80a3410 2024-09-15T03:58:51.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:51.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.059+0000 7f4ed06d4640 10 monclient(hunting): authenticate will time out at 754.703946s 2024-09-15T03:58:51.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecdc6f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec809d4f0 0x7f4ec812e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecdc6f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec809d4f0 0x7f4ec812e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53150/0 (socket says 172.21.15.110:53150) 2024-09-15T03:58:51.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecdc6f640 1 -- 172.21.15.110:0/2871930136 learned_addr learned my addr 172.21.15.110:0/2871930136 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:51.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ec80a3410 0x7f4ec8119160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecdc6f640 10 monclient(hunting): get_auth_request con 0x7f4ec809d4f0 auth_method 0 2024-09-15T03:58:51.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecdc6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecdc6f640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecdc6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 10 monclient(hunting): get_auth_request con 0x7f4ec80a3410 auth_method 0 2024-09-15T03:58:51.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecdc6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecdc6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecdc6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ece470640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ec8069ff0 0x7f4ec809bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 10 monclient(hunting): handle_auth_done global_id 4376 payload 824 2024-09-15T03:58:51.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 1 -- 172.21.15.110:0/2871930136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ec8069ff0 msgr2=0x7f4ec809bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4ec8069ff0 0x7f4ec809bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 1 -- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec809d4f0 msgr2=0x7f4ec812e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec809d4f0 0x7f4ec812e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 10 monclient: _finish_hunting 0 2024-09-15T03:58:51.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 1 monclient: found mon.noname-c 2024-09-15T03:58:51.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:51.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:51.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.060+0000 7f4ecec71640 1 -- 172.21.15.110:0/2871930136 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc009a20 con 0x7f4ec80a3410 2024-09-15T03:58:51.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ecec71640 1 -- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ec80a3410 msgr2=0x7f4ec8119160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:51.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ecec71640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ec80a3410 0x7f4ec8119160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800c490 tx=0x7f4eb8007ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ec67fc640 1 -- 172.21.15.110:0/2871930136 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8005cb0 con 0x7f4ec80a3410 2024-09-15T03:58:51.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ec67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ec67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:51.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ec67fc640 10 monclient: dump: 2024-09-15T03:58:51.078 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.078 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.078 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.078 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.078 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.078 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.078 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.078 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.078 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.079 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ec67fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:51.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ec67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:51.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ed06d4640 5 monclient: authenticate success, global_id 4376 2024-09-15T03:58:51.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ec67fc640 1 -- 172.21.15.110:0/2871930136 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb8005e10 con 0x7f4ec80a3410 2024-09-15T03:58:51.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ec67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:51.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ed06d4640 10 monclient: _renew_subs 2024-09-15T03:58:51.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ec67fc640 1 -- 172.21.15.110:0/2871930136 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8007320 con 0x7f4ec80a3410 2024-09-15T03:58:51.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ed06d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:51.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ed06d4640 1 -- 172.21.15.110:0/2871930136 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec811b590 con 0x7f4ec80a3410 2024-09-15T03:58:51.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ec67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4eccc6d640 4 set_mon_vals no callback set 2024-09-15T03:58:51.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ec67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:51.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ec67fc640 10 monclient: dump: 2024-09-15T03:58:51.080 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.080 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.080 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.080 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.080 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.080 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.080 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.080 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.080 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.081 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ed06d4640 10 monclient: _renew_subs 2024-09-15T03:58:51.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ed06d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:51.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.062+0000 7f4ed06d4640 1 -- 172.21.15.110:0/2871930136 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec811a6a0 con 0x7f4ec80a3410 2024-09-15T03:58:51.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ed06d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:51.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ed06d4640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec812e990 0x7f4ec81196a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ed06d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ed06d4640 1 -- 172.21.15.110:0/2871930136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec8067ab0 con 0x7f4ec812e990 2024-09-15T03:58:51.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ec67fc640 1 -- 172.21.15.110:0/2871930136 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f4eb801b3b0 con 0x7f4ec80a3410 2024-09-15T03:58:51.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ece470640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec812e990 0x7f4ec81196a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ece470640 10 monclient: get_auth_request con 0x7f4ec812e990 auth_method 0 2024-09-15T03:58:51.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ece470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ece470640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ece470640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ece470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ece470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ece470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ec67fc640 1 --2- 172.21.15.110:0/2871930136 >> v2:172.21.15.177:6812/28201 conn(0x7f4eb4038700 0x7f4eb403abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ec67fc640 1 -- 172.21.15.110:0/2871930136 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3530+0+0 (secure 0 0 0) 0x7f4eb804a960 con 0x7f4ec80a3410 2024-09-15T03:58:51.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ece470640 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2024-09-15T03:58:51.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.063+0000 7f4ece470640 1 -- 172.21.15.110:0/2871930136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ec811ba30 con 0x7f4ec812e990 2024-09-15T03:58:51.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.064+0000 7f4ecdc6f640 1 --2- 172.21.15.110:0/2871930136 >> v2:172.21.15.177:6812/28201 conn(0x7f4eb4038700 0x7f4eb403abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.064+0000 7f4ecdc6f640 10 monclient: get_auth_request con 0x7f4eb4038700 auth_method 0 2024-09-15T03:58:51.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.066+0000 7f4ece470640 1 -- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec812e990 msgr2=0x7f4ec81196a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:51.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.066+0000 7f4ece470640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec812e990 0x7f4ec81196a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000d7a0 tx=0x7f4ec0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.066+0000 7f4ec67fc640 1 -- 172.21.15.110:0/2871930136 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0008a20 con 0x7f4ec812e990 2024-09-15T03:58:51.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.066+0000 7f4ec67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.066+0000 7f4ec67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:51.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.066+0000 7f4ec67fc640 10 monclient: dump: 2024-09-15T03:58:51.083 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.083 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.083 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.083 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.083 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.083 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.084 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.084 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.084 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.084 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.066+0000 7f4ec67fc640 1 -- 172.21.15.110:0/2871930136 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4ec000c040 con 0x7f4ec812e990 2024-09-15T03:58:51.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.066+0000 7f4ec67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:51.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.066+0000 7f4ec67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:51.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.066+0000 7f4ec67fc640 1 -- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec812e990 msgr2=0x7f4ec81196a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.066+0000 7f4ec67fc640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec812e990 0x7f4ec81196a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000d7a0 tx=0x7f4ec0008040 comp rx=0 tx=0).stop 2024-09-15T03:58:51.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.066+0000 7f4ec67fc640 1 -- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec812e990 msgr2=0x7f4ec81196a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:58:51.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.066+0000 7f4ec67fc640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec812e990 0x7f4ec81196a0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4ec000d7a0 tx=0x7f4ec0008040 comp rx=0 tx=0).stop 2024-09-15T03:58:51.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.066+0000 7f4ecdc6f640 1 --2- 172.21.15.110:0/2871930136 >> v2:172.21.15.177:6812/28201 conn(0x7f4eb4038700 0x7f4eb403abb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc003440 tx=0x7f4ebc037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.072+0000 7f4ed06d4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:58:51.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.072+0000 7f4ed06d4640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec8119be0 0x7f4ec806c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.072+0000 7f4ed06d4640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.072+0000 7f4ed06d4640 1 -- 172.21.15.110:0/2871930136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec8119fe0 con 0x7f4ec8119be0 2024-09-15T03:58:51.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.072+0000 7f4ece470640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec8119be0 0x7f4ec806c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.072+0000 7f4ece470640 10 monclient: get_auth_request con 0x7f4ec8119be0 auth_method 0 2024-09-15T03:58:51.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.072+0000 7f4ece470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.072+0000 7f4ece470640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.072+0000 7f4ece470640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.072+0000 7f4ece470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.072+0000 7f4ece470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.072+0000 7f4ece470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.072+0000 7f4ece470640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-09-15T03:58:51.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.072+0000 7f4ece470640 1 -- 172.21.15.110:0/2871930136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4ec80647e0 con 0x7f4ec8119be0 2024-09-15T03:58:51.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.074+0000 7f4ece470640 1 -- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec8119be0 msgr2=0x7f4ec806c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:51.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.074+0000 7f4ece470640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec8119be0 0x7f4ec806c610 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0018000 tx=0x7f4ec001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.074+0000 7f4ec67fc640 1 -- 172.21.15.110:0/2871930136 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0020070 con 0x7f4ec8119be0 2024-09-15T03:58:51.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.074+0000 7f4ec67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.074+0000 7f4ec67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:51.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.074+0000 7f4ec67fc640 10 monclient: dump: 2024-09-15T03:58:51.088 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.088 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.088 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.088 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.088 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.088 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.088 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.088 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.089 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.089 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ec67fc640 1 -- 172.21.15.110:0/2871930136 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f4ec00113c0 con 0x7f4ec8119be0 2024-09-15T03:58:51.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ec67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:58:51.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ec67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:51.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ec67fc640 1 -- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec8119be0 msgr2=0x7f4ec806c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ec67fc640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec8119be0 0x7f4ec806c610 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0018000 tx=0x7f4ec001b000 comp rx=0 tx=0).stop 2024-09-15T03:58:51.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ec67fc640 1 -- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec8119be0 msgr2=0x7f4ec806c610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:58:51.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ec67fc640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4ec8119be0 0x7f4ec806c610 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0018000 tx=0x7f4ec001b000 comp rx=0 tx=0).stop 2024-09-15T03:58:51.089 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T03:58:51.089 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T03:58:51.090 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T03:58:51.090 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "leader", 2024-09-15T03:58:51.090 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 8, 2024-09-15T03:58:51.090 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T03:58:51.090 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T03:58:51.090 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T03:58:51.090 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T03:58:51.090 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.090 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 22, 2024-09-15T03:58:51.090 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T03:58:51.090 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T03:58:51.090 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T03:58:51.090 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:58:51.090 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:58:51.091 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:58:51.091 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:58:51.091 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:58:51.091 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:58:51.091 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:58:51.091 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:58:51.091 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:58:51.091 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.091 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:58:51.091 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T03:58:51.091 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:58:51.091 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:58:51.091 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:58:51.091 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:58:51.092 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:58:51.092 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:58:51.092 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:58:51.092 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:58:51.092 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:58:51.092 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.092 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.092 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T03:58:51.092 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T03:58:51.092 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T03:58:51.093 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T03:58:51.093 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T03:58:51.093 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T03:58:51.093 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T03:58:51.093 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T03:58:51.093 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T03:58:51.093 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:58:51.093 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T03:58:51.093 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T03:58:51.093 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T03:58:51.093 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T03:58:51.093 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T03:58:51.093 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T03:58:51.094 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T03:58:51.094 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:58:51.094 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:58:51.094 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:58:51.094 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:58:51.094 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:58:51.094 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:58:51.094 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:58:51.094 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:58:51.094 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:58:51.094 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.094 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T03:58:51.094 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.095 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T03:58:51.095 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.095 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T03:58:51.095 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T03:58:51.095 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:58:51.095 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:58:51.095 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.095 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:58:51.095 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T03:58:51.095 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.095 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.095 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.096 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:58:51.096 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T03:58:51.096 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.096 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.096 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.096 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.096 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T03:58:51.096 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T03:58:51.096 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:58:51.096 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:58:51.097 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:58:51.097 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.097 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.097 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T03:58:51.097 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T03:58:51.097 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:58:51.097 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:58:51.097 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.097 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:58:51.097 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T03:58:51.097 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.097 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.098 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.098 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:58:51.098 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T03:58:51.098 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.098 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.098 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.098 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.098 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T03:58:51.098 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T03:58:51.098 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:58:51.099 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:58:51.099 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:58:51.099 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.099 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.099 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T03:58:51.099 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T03:58:51.099 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:58:51.099 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:58:51.099 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.099 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:58:51.099 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T03:58:51.099 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.099 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.100 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.100 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:58:51.100 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T03:58:51.100 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.100 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.100 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.100 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.100 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T03:58:51.100 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T03:58:51.100 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:58:51.100 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:58:51.101 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:58:51.101 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.101 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.101 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.101 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T03:58:51.101 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T03:58:51.101 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.101 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:58:51.101 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:58:51.101 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T03:58:51.102 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.102 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.102 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T03:58:51.102 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.102 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:58:51.102 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:58:51.102 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T03:58:51.102 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.102 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.102 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T03:58:51.102 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.102 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:58:51.102 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:58:51.103 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T03:58:51.103 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.103 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.103 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.103 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T03:58:51.103 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T03:58:51.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ed06d4640 1 -- 172.21.15.110:0/2871930136 >> v2:172.21.15.177:6812/28201 conn(0x7f4eb4038700 msgr2=0x7f4eb403abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ed06d4640 1 --2- 172.21.15.110:0/2871930136 >> v2:172.21.15.177:6812/28201 conn(0x7f4eb4038700 0x7f4eb403abb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc003440 tx=0x7f4ebc037000 comp rx=0 tx=0).stop 2024-09-15T03:58:51.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ed06d4640 10 monclient: shutdown 2024-09-15T03:58:51.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ed06d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:51.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ed06d4640 1 -- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ec80a3410 msgr2=0x7f4ec8119160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ed06d4640 1 --2- 172.21.15.110:0/2871930136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4ec80a3410 0x7f4ec8119160 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800c490 tx=0x7f4eb8007ee0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ecec71640 1 -- 172.21.15.110:0/2871930136 reap_dead start 2024-09-15T03:58:51.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ecec71640 1 -- 172.21.15.110:0/2871930136 reap_dead start 2024-09-15T03:58:51.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ed06d4640 1 -- 172.21.15.110:0/2871930136 shutdown_connections 2024-09-15T03:58:51.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ed06d4640 1 -- 172.21.15.110:0/2871930136 >> 172.21.15.110:0/2871930136 conn(0x7f4ec80635b0 msgr2=0x7f4ec80a2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:51.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ed06d4640 1 -- 172.21.15.110:0/2871930136 shutdown_connections 2024-09-15T03:58:51.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.075+0000 7f4ed06d4640 1 -- 172.21.15.110:0/2871930136 wait complete. 2024-09-15T03:58:51.106 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:58:51.106 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:58:51.106 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:58:51.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.198+0000 7fd6d7d9d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:51.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.198+0000 7fd6d7d9d640 10 monclient: build_initial_monmap 2024-09-15T03:58:51.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.198+0000 7fd6d7d9d640 10 monclient: monmap: 2024-09-15T03:58:51.202 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:51.203 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.203 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:51.199290+0000 2024-09-15T03:58:51.203 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:51.199290+0000 2024-09-15T03:58:51.203 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:51.203 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.203 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:51.203 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:51.203 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:51.203 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:51.203 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:51.203 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.198+0000 7fd6d7d9d640 1 Processor -- start 2024-09-15T03:58:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 1 -- start start 2024-09-15T03:58:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 10 monclient: init 2024-09-15T03:58:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-15T03:58:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6d00a1a10 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd6d009ee80 0x7fd6d0131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd6d009ee80 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 0x7fd6d0138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6d01369e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d009f2c0 con 0x7fd6d00a1a10 2024-09-15T03:58:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d00a0850 con 0x7fd6d009ee80 2024-09-15T03:58:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d009c9a0 con 0x7fd6d01369e0 2024-09-15T03:58:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d7d9d640 10 monclient(hunting): authenticate will time out at 754.843533s 2024-09-15T03:58:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 0x7fd6d0138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d5338640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd6d009ee80 0x7fd6d0131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 0x7fd6d0138dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43584/0 (socket says 172.21.15.110:43584) 2024-09-15T03:58:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 1 -- 172.21.15.110:0/1622906636 learned_addr learned my addr 172.21.15.110:0/1622906636 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 10 monclient(hunting): get_auth_request con 0x7fd6d01369e0 auth_method 0 2024-09-15T03:58:51.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d5338640 10 monclient(hunting): get_auth_request con 0x7fd6d009ee80 auth_method 0 2024-09-15T03:58:51.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d5338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d5338640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d5338640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d5338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d5338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d5338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 10 monclient(hunting): handle_auth_done global_id 4379 payload 274 2024-09-15T03:58:51.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 1 -- 172.21.15.110:0/1622906636 >> v2:172.21.15.110:3300/0 conn(0x7fd6d009ee80 msgr2=0x7fd6d0131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:51.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 1 --2- 172.21.15.110:0/1622906636 >> v2:172.21.15.110:3300/0 conn(0x7fd6d009ee80 0x7fd6d0131b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 1 -- 172.21.15.110:0/1622906636 >> v1:172.21.15.177:6789/0 conn(0x7fd6d00a1a10 legacy=0x7fd6d009e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:51.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 10 monclient: _finish_hunting 0 2024-09-15T03:58:51.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 1 monclient: found mon.noname-b 2024-09-15T03:58:51.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:51.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:51.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d633a640 1 -- 172.21.15.110:0/1622906636 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d009cc80 con 0x7fd6d01369e0 2024-09-15T03:58:51.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.199+0000 7fd6d5338640 1 --2- 172.21.15.110:0/1622906636 >> v2:172.21.15.110:3300/0 conn(0x7fd6d009ee80 0x7fd6d0131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:51.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d633a640 1 -- 172.21.15.110:0/1622906636 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 msgr2=0x7fd6d0138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:51.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d633a640 1 --2- 172.21.15.110:0/1622906636 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 0x7fd6d0138dc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009f20 tx=0x7fd6c8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92e607acac6ddf96 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d4b37640 1 -- 172.21.15.110:0/1622906636 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8007610 con 0x7fd6d01369e0 2024-09-15T03:58:51.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d4b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d4b37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:51.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d4b37640 10 monclient: dump: 2024-09-15T03:58:51.224 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.225 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.225 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.225 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.225 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.225 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.225 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.225 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.225 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.225 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d4b37640 10 monclient: _finish_auth 0 2024-09-15T03:58:51.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d4b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:51.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d4b37640 1 -- 172.21.15.110:0/1622906636 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c802d050 con 0x7fd6d01369e0 2024-09-15T03:58:51.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d7d9d640 5 monclient: authenticate success, global_id 4379 2024-09-15T03:58:51.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d7d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:51.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d4b37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:51.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d7d9d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:51.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d4b37640 1 -- 172.21.15.110:0/1622906636 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80079e0 con 0x7fd6d01369e0 2024-09-15T03:58:51.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d4b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d7d9d640 4 set_mon_vals no callback set 2024-09-15T03:58:51.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d4b37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:51.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.202+0000 7fd6d4b37640 10 monclient: dump: 2024-09-15T03:58:51.226 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.226 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.227 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.227 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.227 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.227 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.227 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.227 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.227 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.227 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.203+0000 7fd6d7d9d640 10 monclient: shutdown 2024-09-15T03:58:51.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.203+0000 7fd6d7d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:51.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.203+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/1622906636 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 msgr2=0x7fd6d0138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.203+0000 7fd6d7d9d640 1 --2- 172.21.15.110:0/1622906636 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 0x7fd6d0138dc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009f20 tx=0x7fd6c8009840 comp rx=0 tx=0).stop 2024-09-15T03:58:51.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.203+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/1622906636 shutdown_connections 2024-09-15T03:58:51.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.203+0000 7fd6d7d9d640 1 --2- 172.21.15.110:0/1622906636 >> v2:172.21.15.110:3300/0 conn(0x7fd6d009ee80 0x7fd6d0131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.203+0000 7fd6d7d9d640 1 --2- 172.21.15.110:0/1622906636 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 0x7fd6d0138dc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.203+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/1622906636 >> 172.21.15.110:0/1622906636 conn(0x7fd6d006e860 msgr2=0x7fd6d006ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:51.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.203+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/1622906636 shutdown_connections 2024-09-15T03:58:51.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.203+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/1622906636 wait complete. 2024-09-15T03:58:51.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 10 monclient: build_initial_monmap 2024-09-15T03:58:51.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 10 monclient: monmap: 2024-09-15T03:58:51.228 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:51.228 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:51.228 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:51.229 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 1 Processor -- start 2024-09-15T03:58:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 1 -- start start 2024-09-15T03:58:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 10 monclient: init 2024-09-15T03:58:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:51.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:58:51.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6d009ee80 0x7fd6d01361a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd6d009ee80 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:51.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6d00a1a10 0x7fd6d012fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd6d00a1a10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:51.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 0x7fd6d0130220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd6d01369e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:51.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d0133150 con 0x7fd6d009ee80 2024-09-15T03:58:51.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d0134f40 con 0x7fd6d00a1a10 2024-09-15T03:58:51.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d0132790 con 0x7fd6d01369e0 2024-09-15T03:58:51.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:51.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d7d9d640 10 monclient(hunting): authenticate will time out at 754.848805s 2024-09-15T03:58:51.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d633a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 0x7fd6d0130220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d5b39640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6d009ee80 0x7fd6d01361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d633a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 0x7fd6d0130220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43586/0 (socket says 172.21.15.110:43586) 2024-09-15T03:58:51.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d633a640 1 -- 172.21.15.110:0/543651979 learned_addr learned my addr 172.21.15.110:0/543651979 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:51.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d5b39640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6d009ee80 0x7fd6d01361a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53178/0 (socket says 172.21.15.110:53178) 2024-09-15T03:58:51.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d633a640 10 monclient(hunting): get_auth_request con 0x7fd6d01369e0 auth_method 0 2024-09-15T03:58:51.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d633a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d633a640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d633a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d5b39640 10 monclient(hunting): get_auth_request con 0x7fd6d009ee80 auth_method 0 2024-09-15T03:58:51.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d5b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d5b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.204+0000 7fd6d5b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d633a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d633a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d633a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d5b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d5b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d5b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d5338640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6d00a1a10 0x7fd6d012fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d633a640 10 monclient(hunting): handle_auth_done global_id 4382 payload 824 2024-09-15T03:58:51.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d633a640 1 -- 172.21.15.110:0/543651979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6d00a1a10 msgr2=0x7fd6d012fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d633a640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6d00a1a10 0x7fd6d012fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d633a640 1 -- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6d009ee80 msgr2=0x7fd6d01361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d633a640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6d009ee80 0x7fd6d01361a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d633a640 10 monclient: _finish_hunting 0 2024-09-15T03:58:51.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d633a640 1 monclient: found mon.noname-c 2024-09-15T03:58:51.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d633a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:51.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d633a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:51.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d633a640 1 -- 172.21.15.110:0/543651979 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c0009390 con 0x7fd6d01369e0 2024-09-15T03:58:51.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.205+0000 7fd6d5b39640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd6d009ee80 0x7fd6d01361a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:58:51.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6d633a640 1 -- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 msgr2=0x7fd6d0130220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:51.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6d633a640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 0x7fd6d0130220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009f80 tx=0x7fd6c8036000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6c5ffb640 1 -- 172.21.15.110:0/543651979 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8007170 con 0x7fd6d01369e0 2024-09-15T03:58:51.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6c5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:51.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6c5ffb640 10 monclient: dump: 2024-09-15T03:58:51.235 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.235 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.235 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.236 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.236 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.236 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.236 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.236 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.236 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.236 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6c5ffb640 10 monclient: _finish_auth 0 2024-09-15T03:58:51.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:51.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6d7d9d640 5 monclient: authenticate success, global_id 4382 2024-09-15T03:58:51.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6c5ffb640 1 -- 172.21.15.110:0/543651979 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c8032cc0 con 0x7fd6d01369e0 2024-09-15T03:58:51.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6c5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:51.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6d7d9d640 10 monclient: _renew_subs 2024-09-15T03:58:51.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6c5ffb640 1 -- 172.21.15.110:0/543651979 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c803b680 con 0x7fd6d01369e0 2024-09-15T03:58:51.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6d7d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:51.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/543651979 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c8009ae0 con 0x7fd6d01369e0 2024-09-15T03:58:51.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:51.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6c7fff640 4 set_mon_vals no callback set 2024-09-15T03:58:51.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6c5ffb640 10 monclient: dump: 2024-09-15T03:58:51.237 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.237 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.237 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.238 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.238 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.238 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.238 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.238 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.238 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.238 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6d7d9d640 10 monclient: _renew_subs 2024-09-15T03:58:51.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6d7d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:51.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.207+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/543651979 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d0134300 con 0x7fd6d01369e0 2024-09-15T03:58:51.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d7d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:51.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6c5ffb640 1 -- 172.21.15.110:0/543651979 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fd6c8038050 con 0x7fd6d01369e0 2024-09-15T03:58:51.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d7d9d640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0133320 0x7fd6d0130760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d7d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/543651979 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d0063e80 con 0x7fd6d0133320 2024-09-15T03:58:51.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d5b39640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0133320 0x7fd6d0130760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d5b39640 10 monclient: get_auth_request con 0x7fd6d0133320 auth_method 0 2024-09-15T03:58:51.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d5b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d5b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d5b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6c5ffb640 1 --2- 172.21.15.110:0/543651979 >> v2:172.21.15.177:6812/28201 conn(0x7fd6b4038700 0x7fd6b403abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d5b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d5b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d5b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6c5ffb640 1 -- 172.21.15.110:0/543651979 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fd6c806c280 con 0x7fd6d01369e0 2024-09-15T03:58:51.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d5b39640 10 monclient(hunting): handle_auth_done global_id 4385 payload 824 2024-09-15T03:58:51.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d5b39640 1 -- 172.21.15.110:0/543651979 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6d00647e0 con 0x7fd6d0133320 2024-09-15T03:58:51.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.208+0000 7fd6d5338640 1 --2- 172.21.15.110:0/543651979 >> v2:172.21.15.177:6812/28201 conn(0x7fd6b4038700 0x7fd6b403abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.209+0000 7fd6d5338640 10 monclient: get_auth_request con 0x7fd6b4038700 auth_method 0 2024-09-15T03:58:51.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.211+0000 7fd6d5b39640 1 -- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0133320 msgr2=0x7fd6d0130760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:51.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.211+0000 7fd6d5b39640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0133320 0x7fd6d0130760 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00eb00 tx=0x7fd6cc009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.211+0000 7fd6c5ffb640 1 -- 172.21.15.110:0/543651979 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc00ce80 con 0x7fd6d0133320 2024-09-15T03:58:51.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.211+0000 7fd6c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.211+0000 7fd6c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:51.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.211+0000 7fd6c5ffb640 10 monclient: dump: 2024-09-15T03:58:51.240 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.240 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.240 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.241 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.241 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.241 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.241 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.241 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.241 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.241 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.211+0000 7fd6c5ffb640 1 -- 172.21.15.110:0/543651979 <== mon.2 v2:172.21.15.110:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fd6cc0123f0 con 0x7fd6d0133320 2024-09-15T03:58:51.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.211+0000 7fd6c5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:51.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.211+0000 7fd6c5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:51.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.211+0000 7fd6c5ffb640 1 -- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0133320 msgr2=0x7fd6d0130760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.211+0000 7fd6c5ffb640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0133320 0x7fd6d0130760 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00eb00 tx=0x7fd6cc009060 comp rx=0 tx=0).stop 2024-09-15T03:58:51.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.211+0000 7fd6c5ffb640 1 -- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0133320 msgr2=0x7fd6d0130760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:58:51.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.211+0000 7fd6c5ffb640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0133320 0x7fd6d0130760 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00eb00 tx=0x7fd6cc009060 comp rx=0 tx=0).stop 2024-09-15T03:58:51.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.211+0000 7fd6d5338640 1 --2- 172.21.15.110:0/543651979 >> v2:172.21.15.177:6812/28201 conn(0x7fd6b4038700 0x7fd6b403abb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0006e70 tx=0x7fd6c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.217+0000 7fd6d7d9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:58:51.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.217+0000 7fd6d7d9d640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0136db0 0x7fd6d01371b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.217+0000 7fd6d7d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.217+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/543651979 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d0067130 con 0x7fd6d0136db0 2024-09-15T03:58:51.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.217+0000 7fd6d5b39640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0136db0 0x7fd6d01371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.217+0000 7fd6d5b39640 10 monclient: get_auth_request con 0x7fd6d0136db0 auth_method 0 2024-09-15T03:58:51.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.217+0000 7fd6d5b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.217+0000 7fd6d5b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.217+0000 7fd6d5b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.217+0000 7fd6d5b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.217+0000 7fd6d5b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.217+0000 7fd6d5b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.217+0000 7fd6d5b39640 10 monclient(hunting): handle_auth_done global_id 4388 payload 824 2024-09-15T03:58:51.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.217+0000 7fd6d5b39640 1 -- 172.21.15.110:0/543651979 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd6d01356b0 con 0x7fd6d0136db0 2024-09-15T03:58:51.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.219+0000 7fd6d5b39640 1 -- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0136db0 msgr2=0x7fd6d01371b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:51.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.219+0000 7fd6d5b39640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0136db0 0x7fd6d01371b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc017000 tx=0x7fd6cc015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.219+0000 7fd6c5ffb640 1 -- 172.21.15.110:0/543651979 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc01cde0 con 0x7fd6d0136db0 2024-09-15T03:58:51.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.219+0000 7fd6c5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.219+0000 7fd6c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:51.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.219+0000 7fd6c5ffb640 10 monclient: dump: 2024-09-15T03:58:51.243 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.244 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.244 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.244 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.244 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.244 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.244 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.244 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.244 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.244 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6c5ffb640 1 -- 172.21.15.110:0/543651979 <== mon.2 v2:172.21.15.110:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7fd6cc00cc70 con 0x7fd6d0136db0 2024-09-15T03:58:51.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6c5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:58:51.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6c5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:51.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6c5ffb640 1 -- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0136db0 msgr2=0x7fd6d01371b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6c5ffb640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0136db0 0x7fd6d01371b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc017000 tx=0x7fd6cc015000 comp rx=0 tx=0).stop 2024-09-15T03:58:51.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6c5ffb640 1 -- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0136db0 msgr2=0x7fd6d01371b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:58:51.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6c5ffb640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d0136db0 0x7fd6d01371b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.245 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T03:58:51.245 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T03:58:51.245 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T03:58:51.245 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T03:58:51.246 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 8, 2024-09-15T03:58:51.246 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T03:58:51.246 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T03:58:51.246 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T03:58:51.246 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T03:58:51.246 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.246 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 22, 2024-09-15T03:58:51.246 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T03:58:51.246 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T03:58:51.246 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T03:58:51.246 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:58:51.247 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:58:51.247 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:58:51.247 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:58:51.247 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:58:51.247 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:58:51.247 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:58:51.247 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:58:51.247 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:58:51.247 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.247 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:58:51.247 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T03:58:51.247 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:58:51.248 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:58:51.248 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:58:51.248 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:58:51.248 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:58:51.248 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:58:51.248 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:58:51.248 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:58:51.248 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:58:51.248 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.249 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T03:58:51.249 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T03:58:51.249 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T03:58:51.249 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T03:58:51.249 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T03:58:51.249 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T03:58:51.249 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T03:58:51.249 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T03:58:51.249 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T03:58:51.249 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:58:51.249 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T03:58:51.250 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T03:58:51.250 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T03:58:51.250 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T03:58:51.250 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T03:58:51.250 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T03:58:51.250 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T03:58:51.250 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:58:51.250 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:58:51.250 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:58:51.250 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:58:51.250 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:58:51.250 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:58:51.250 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:58:51.251 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:58:51.251 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:58:51.251 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.251 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T03:58:51.251 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.251 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T03:58:51.251 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.251 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T03:58:51.251 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T03:58:51.251 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:58:51.251 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:58:51.251 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.251 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:58:51.251 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T03:58:51.252 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.252 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.252 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.252 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:58:51.252 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T03:58:51.252 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.252 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.252 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.252 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.252 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T03:58:51.253 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T03:58:51.253 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:58:51.253 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:58:51.253 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:58:51.253 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.253 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.253 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T03:58:51.253 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T03:58:51.253 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:58:51.253 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:58:51.253 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.253 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:58:51.253 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T03:58:51.254 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.254 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.254 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.254 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:58:51.254 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T03:58:51.254 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.254 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.254 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.254 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.254 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T03:58:51.254 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T03:58:51.254 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:58:51.254 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:58:51.254 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:58:51.255 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.255 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.255 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T03:58:51.255 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T03:58:51.255 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:58:51.255 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:58:51.255 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.255 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:58:51.255 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T03:58:51.255 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.255 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.255 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.255 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:58:51.256 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T03:58:51.256 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.256 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.256 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.256 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.256 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T03:58:51.256 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T03:58:51.256 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:58:51.256 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:58:51.256 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:58:51.256 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.256 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.256 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.257 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T03:58:51.257 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T03:58:51.257 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.257 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:58:51.257 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:58:51.257 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T03:58:51.257 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.257 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.257 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T03:58:51.257 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.257 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:58:51.257 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:58:51.258 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 2 2024-09-15T03:58:51.258 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.258 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.258 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T03:58:51.258 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.258 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:58:51.258 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:58:51.258 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 3 2024-09-15T03:58:51.258 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.259 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.259 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.259 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T03:58:51.259 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T03:58:51.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/543651979 >> v2:172.21.15.177:6812/28201 conn(0x7fd6b4038700 msgr2=0x7fd6b403abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6d7d9d640 1 --2- 172.21.15.110:0/543651979 >> v2:172.21.15.177:6812/28201 conn(0x7fd6b4038700 0x7fd6b403abb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0006e70 tx=0x7fd6c0008040 comp rx=0 tx=0).stop 2024-09-15T03:58:51.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6d7d9d640 10 monclient: shutdown 2024-09-15T03:58:51.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6d7d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:51.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 msgr2=0x7fd6d0130220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6d633a640 1 -- 172.21.15.110:0/543651979 reap_dead start 2024-09-15T03:58:51.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6d7d9d640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 0x7fd6d0130220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009f80 tx=0x7fd6c8036000 comp rx=0 tx=0).stop 2024-09-15T03:58:51.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/543651979 shutdown_connections 2024-09-15T03:58:51.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6d7d9d640 1 --2- 172.21.15.110:0/543651979 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd6d01369e0 0x7fd6d0130220 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/543651979 >> 172.21.15.110:0/543651979 conn(0x7fd6d006e860 msgr2=0x7fd6d00a4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:51.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/543651979 shutdown_connections 2024-09-15T03:58:51.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.220+0000 7fd6d7d9d640 1 -- 172.21.15.110:0/543651979 wait complete. 2024-09-15T03:58:51.263 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:58:51.263 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:58:51.263 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:58:51.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.356+0000 7f78fea17640 10 monclient: get_monmap_and_config 2024-09-15T03:58:51.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.356+0000 7f78fea17640 10 monclient: build_initial_monmap 2024-09-15T03:58:51.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.356+0000 7f78fea17640 10 monclient: monmap: 2024-09-15T03:58:51.357 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:51.357 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.358 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:51.357437+0000 2024-09-15T03:58:51.358 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:51.357437+0000 2024-09-15T03:58:51.358 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:51.358 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.358 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:51.358 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:51.358 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:51.358 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:51.358 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:51.359 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 1 Processor -- start 2024-09-15T03:58:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 1 -- start start 2024-09-15T03:58:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 10 monclient: init 2024-09-15T03:58:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-15T03:58:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78f809b0d0 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78f8134290 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78f8137d90 0x7f78f813a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 10 monclient(hunting): picked mon.noname-b con 0x7f78f8137d90 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f80a1330 con 0x7f78f8134290 2024-09-15T03:58:51.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f806f000 con 0x7f78f809b0d0 2024-09-15T03:58:51.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f809a290 con 0x7f78f8137d90 2024-09-15T03:58:51.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:51.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fea17640 10 monclient(hunting): authenticate will time out at 755.001724s 2024-09-15T03:58:51.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fcfb4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78f8137d90 0x7f78f813a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78f77fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f78f8134290 0x7f78f8136670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:34166/0 (socket says 172.21.15.110:34166) 2024-09-15T03:58:51.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78f77fe640 1 -- 172.21.15.110:0/1567386666 learned_addr learned my addr 172.21.15.110:0/1567386666 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:51.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fcfb4640 10 monclient(hunting): get_auth_request con 0x7f78f8137d90 auth_method 0 2024-09-15T03:58:51.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fcfb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fcfb4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fcfb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fcfb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fcfb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.357+0000 7f78fcfb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.358+0000 7f78fcfb4640 10 monclient(hunting): handle_auth_done global_id 4391 payload 274 2024-09-15T03:58:51.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.358+0000 7f78fcfb4640 1 -- 172.21.15.110:0/1567386666 >> v1:172.21.15.177:6789/0 conn(0x7f78f809b0d0 legacy=0x7f78f806e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:51.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.358+0000 7f78fcfb4640 1 -- 172.21.15.110:0/1567386666 >> v1:172.21.15.110:6789/0 conn(0x7f78f8134290 legacy=0x7f78f8136670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:51.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.358+0000 7f78fcfb4640 10 monclient: _finish_hunting 0 2024-09-15T03:58:51.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.358+0000 7f78fcfb4640 1 monclient: found mon.noname-b 2024-09-15T03:58:51.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.358+0000 7f78fcfb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:51.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.358+0000 7f78fcfb4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:51.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.358+0000 7f78fcfb4640 1 -- 172.21.15.110:0/1567386666 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f8099fb0 con 0x7f78f8137d90 2024-09-15T03:58:51.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.360+0000 7f78fcfb4640 1 -- 172.21.15.110:0/1567386666 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78f8137d90 msgr2=0x7f78f813a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:51.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.360+0000 7f78fcfb4640 1 --2- 172.21.15.110:0/1567386666 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78f8137d90 0x7f78f813a170 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78f0009e90 tx=0x7f78f0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=633a464180cd8176 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78f6ffd640 1 -- 172.21.15.110:0/1567386666 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f0007840 con 0x7f78f8137d90 2024-09-15T03:58:51.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:51.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78f6ffd640 10 monclient: dump: 2024-09-15T03:58:51.370 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.370 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.370 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.370 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.370 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.371 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78f6ffd640 10 monclient: _finish_auth 0 2024-09-15T03:58:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78f6ffd640 1 -- 172.21.15.110:0/1567386666 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f002d050 con 0x7f78f8137d90 2024-09-15T03:58:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78fea17640 5 monclient: authenticate success, global_id 4391 2024-09-15T03:58:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78fea17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78fea17640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78f6ffd640 1 -- 172.21.15.110:0/1567386666 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f0007bd0 con 0x7f78f8137d90 2024-09-15T03:58:51.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78fea17640 4 set_mon_vals no callback set 2024-09-15T03:58:51.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:51.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78f6ffd640 10 monclient: dump: 2024-09-15T03:58:51.372 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.372 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.372 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.372 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.372 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.372 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.372 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.373 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78fea17640 10 monclient: shutdown 2024-09-15T03:58:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78fea17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78fea17640 1 -- 172.21.15.110:0/1567386666 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78f8137d90 msgr2=0x7f78f813a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78fea17640 1 --2- 172.21.15.110:0/1567386666 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78f8137d90 0x7f78f813a170 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78f0009e90 tx=0x7f78f0009780 comp rx=0 tx=0).stop 2024-09-15T03:58:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78fea17640 1 -- 172.21.15.110:0/1567386666 shutdown_connections 2024-09-15T03:58:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78fea17640 1 --2- 172.21.15.110:0/1567386666 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78f8137d90 0x7f78f813a170 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78fea17640 1 -- 172.21.15.110:0/1567386666 >> 172.21.15.110:0/1567386666 conn(0x7f78f809e570 msgr2=0x7f78f809e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:51.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78fea17640 1 -- 172.21.15.110:0/1567386666 shutdown_connections 2024-09-15T03:58:51.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.361+0000 7f78fea17640 1 -- 172.21.15.110:0/1567386666 wait complete. 2024-09-15T03:58:51.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.362+0000 7f78fea17640 10 monclient: build_initial_monmap 2024-09-15T03:58:51.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.362+0000 7f78fea17640 10 monclient: monmap: 2024-09-15T03:58:51.374 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:51.374 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:51.374 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:51.374 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:51.374 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:51.374 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.374 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:51.374 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:51.374 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:51.374 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.362+0000 7f78fea17640 1 Processor -- start 2024-09-15T03:58:51.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.362+0000 7f78fea17640 1 -- start start 2024-09-15T03:58:51.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.362+0000 7f78fea17640 10 monclient: init 2024-09-15T03:58:51.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:51.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:58:51.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f809b0d0 0x7f78f81411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 10 monclient(hunting): picked mon.noname-b con 0x7f78f809b0d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:51.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78f8134290 0x7f78f8141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 10 monclient(hunting): picked mon.noname-a con 0x7f78f8134290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:51.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78f8137d90 0x7f78f8141c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 10 monclient(hunting): picked mon.noname-c con 0x7f78f8137d90 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:51.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f809d4f0 con 0x7f78f8134290 2024-09-15T03:58:51.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f80a1690 con 0x7f78f809b0d0 2024-09-15T03:58:51.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f811c630 con 0x7f78f8137d90 2024-09-15T03:58:51.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:51.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fea17640 10 monclient(hunting): authenticate will time out at 755.007212s 2024-09-15T03:58:51.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78f8134290 0x7f78f8141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fcfb4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78f8137d90 0x7f78f8141c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78f8134290 0x7f78f8141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53192/0 (socket says 172.21.15.110:53192) 2024-09-15T03:58:51.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 1 -- 172.21.15.110:0/3173748210 learned_addr learned my addr 172.21.15.110:0/3173748210 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 10 monclient(hunting): get_auth_request con 0x7f78f8134290 auth_method 0 2024-09-15T03:58:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fcfb4640 10 monclient(hunting): get_auth_request con 0x7f78f8137d90 auth_method 0 2024-09-15T03:58:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fcfb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fcfb4640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fcfb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fcfb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fcfb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78fcfb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f7fff640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f809b0d0 0x7f78f81411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f7fff640 10 monclient(hunting): get_auth_request con 0x7f78f809b0d0 auth_method 0 2024-09-15T03:58:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2024-09-15T03:58:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 1 -- 172.21.15.110:0/3173748210 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78f8137d90 msgr2=0x7f78f8141c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f78f8137d90 0x7f78f8141c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 1 -- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f809b0d0 msgr2=0x7f78f81411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f809b0d0 0x7f78f81411f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 10 monclient: _finish_hunting 0 2024-09-15T03:58:51.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 1 monclient: found mon.noname-a 2024-09-15T03:58:51.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:51.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T03:58:51.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f77fe640 1 -- 172.21.15.110:0/3173748210 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f0009a20 con 0x7f78f8134290 2024-09-15T03:58:51.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.363+0000 7f78f7fff640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f809b0d0 0x7f78f81411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:58:51.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78f77fe640 1 -- 172.21.15.110:0/3173748210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78f8134290 msgr2=0x7f78f8141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:51.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78f77fe640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78f8134290 0x7f78f8141730 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f78ec00c5e0 tx=0x7f78ec004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78dffff640 1 -- 172.21.15.110:0/3173748210 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ec005cb0 con 0x7f78f8134290 2024-09-15T03:58:51.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:58:51.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78dffff640 10 monclient: dump: 2024-09-15T03:58:51.379 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.380 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.380 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.380 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.380 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.380 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.380 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.380 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.381 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.381 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78dffff640 10 monclient: _finish_auth 0 2024-09-15T03:58:51.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:51.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78fea17640 5 monclient: authenticate success, global_id 4359 2024-09-15T03:58:51.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78dffff640 1 -- 172.21.15.110:0/3173748210 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ec005e30 con 0x7f78f8134290 2024-09-15T03:58:51.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78dffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:51.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78dffff640 1 -- 172.21.15.110:0/3173748210 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ec007320 con 0x7f78f8134290 2024-09-15T03:58:51.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:58:51.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78dffff640 10 monclient: dump: 2024-09-15T03:58:51.381 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.382 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.382 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.382 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.382 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.382 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.382 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.382 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.382 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.382 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78fea17640 10 monclient: _renew_subs 2024-09-15T03:58:51.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78fea17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:51.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78fea17640 1 -- 172.21.15.110:0/3173748210 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f811cb40 con 0x7f78f8134290 2024-09-15T03:58:51.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.365+0000 7f78f67fc640 4 set_mon_vals no callback set 2024-09-15T03:58:51.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.366+0000 7f78fea17640 10 monclient: _renew_subs 2024-09-15T03:58:51.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.366+0000 7f78fea17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T03:58:51.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.366+0000 7f78fea17640 1 -- 172.21.15.110:0/3173748210 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f811d2a0 con 0x7f78f8134290 2024-09-15T03:58:51.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.366+0000 7f78fea17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:51.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.366+0000 7f78fea17640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806bbd0 0x7f78f806bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.366+0000 7f78fea17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.366+0000 7f78fea17640 1 -- 172.21.15.110:0/3173748210 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f8068c10 con 0x7f78f806bbd0 2024-09-15T03:58:51.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.366+0000 7f78dffff640 1 -- 172.21.15.110:0/3173748210 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f78ec01b3b0 con 0x7f78f8134290 2024-09-15T03:58:51.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.367+0000 7f78dffff640 1 --2- 172.21.15.110:0/3173748210 >> v2:172.21.15.177:6812/28201 conn(0x7f78e4038700 0x7f78e403abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.367+0000 7f78f7fff640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806bbd0 0x7f78f806bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.367+0000 7f78f7fff640 10 monclient: get_auth_request con 0x7f78f806bbd0 auth_method 0 2024-09-15T03:58:51.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.367+0000 7f78f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.367+0000 7f78f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.367+0000 7f78f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.367+0000 7f78dffff640 1 -- 172.21.15.110:0/3173748210 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f78ec04ac60 con 0x7f78f8134290 2024-09-15T03:58:51.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.367+0000 7f78f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.367+0000 7f78f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.367+0000 7f78f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.367+0000 7f78fcfb4640 1 --2- 172.21.15.110:0/3173748210 >> v2:172.21.15.177:6812/28201 conn(0x7f78e4038700 0x7f78e403abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.367+0000 7f78fcfb4640 10 monclient: get_auth_request con 0x7f78e4038700 auth_method 0 2024-09-15T03:58:51.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.367+0000 7f78f7fff640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2024-09-15T03:58:51.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.367+0000 7f78f7fff640 1 -- 172.21.15.110:0/3173748210 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78f80647e0 con 0x7f78f806bbd0 2024-09-15T03:58:51.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.369+0000 7f78f7fff640 1 -- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806bbd0 msgr2=0x7f78f806bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:51.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.369+0000 7f78f7fff640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806bbd0 0x7f78f806bfa0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f78e800e710 tx=0x7f78e8009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.369+0000 7f78fcfb4640 1 --2- 172.21.15.110:0/3173748210 >> v2:172.21.15.177:6812/28201 conn(0x7f78e4038700 0x7f78e403abb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f78f806ae00 tx=0x7f78f0037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.369+0000 7f78dffff640 1 -- 172.21.15.110:0/3173748210 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e800c820 con 0x7f78f806bbd0 2024-09-15T03:58:51.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.369+0000 7f78dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.369+0000 7f78dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:51.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.369+0000 7f78dffff640 10 monclient: dump: 2024-09-15T03:58:51.388 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.388 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.388 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.388 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.388 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.389 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.389 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.389 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.389 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.389 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.370+0000 7f78dffff640 1 -- 172.21.15.110:0/3173748210 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f78e800c980 con 0x7f78f806bbd0 2024-09-15T03:58:51.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.370+0000 7f78dffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:51.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.370+0000 7f78dffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:51.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.370+0000 7f78dffff640 1 -- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806bbd0 msgr2=0x7f78f806bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.370+0000 7f78dffff640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806bbd0 0x7f78f806bfa0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f78e800e710 tx=0x7f78e8009040 comp rx=0 tx=0).stop 2024-09-15T03:58:51.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.370+0000 7f78dffff640 1 -- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806bbd0 msgr2=0x7f78f806bfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:58:51.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.370+0000 7f78dffff640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806bbd0 0x7f78f806bfa0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.375+0000 7f78fea17640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:58:51.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.375+0000 7f78fea17640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806c4e0 0x7f78f811ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.375+0000 7f78fea17640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.375+0000 7f78fea17640 1 -- 172.21.15.110:0/3173748210 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f80a0760 con 0x7f78f806c4e0 2024-09-15T03:58:51.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.376+0000 7f78f7fff640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806c4e0 0x7f78f811ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.376+0000 7f78f7fff640 10 monclient: get_auth_request con 0x7f78f806c4e0 auth_method 0 2024-09-15T03:58:51.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.376+0000 7f78f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.376+0000 7f78f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.376+0000 7f78f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.376+0000 7f78f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.390 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.376+0000 7f78f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.376+0000 7f78f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.376+0000 7f78f7fff640 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2024-09-15T03:58:51.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.376+0000 7f78f7fff640 1 -- 172.21.15.110:0/3173748210 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f78f811f4b0 con 0x7f78f806c4e0 2024-09-15T03:58:51.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.378+0000 7f78f7fff640 1 -- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806c4e0 msgr2=0x7f78f811ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:51.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.378+0000 7f78f7fff640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806c4e0 0x7f78f811ab80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f78e800efd0 tx=0x7f78e801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78dffff640 1 -- 172.21.15.110:0/3173748210 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e8022480 con 0x7f78f806c4e0 2024-09-15T03:58:51.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:51.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78dffff640 10 monclient: dump: 2024-09-15T03:58:51.391 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.392 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.392 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.392 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.392 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.392 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.392 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.392 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.392 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.392 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78dffff640 1 -- 172.21.15.110:0/3173748210 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f78e800c610 con 0x7f78f806c4e0 2024-09-15T03:58:51.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78dffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:58:51.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78dffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:51.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78dffff640 1 -- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806c4e0 msgr2=0x7f78f811ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78dffff640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806c4e0 0x7f78f811ab80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f78e800efd0 tx=0x7f78e801a000 comp rx=0 tx=0).stop 2024-09-15T03:58:51.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78dffff640 1 -- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806c4e0 msgr2=0x7f78f811ab80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:58:51.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78dffff640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f78f806c4e0 0x7f78f811ab80 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f78e800efd0 tx=0x7f78e801a000 comp rx=0 tx=0).stop 2024-09-15T03:58:51.393 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T03:58:51.393 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T03:58:51.393 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T03:58:51.393 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T03:58:51.394 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 8, 2024-09-15T03:58:51.394 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T03:58:51.394 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T03:58:51.394 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T03:58:51.394 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T03:58:51.394 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.394 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 23, 2024-09-15T03:58:51.394 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T03:58:51.394 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T03:58:51.394 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T03:58:51.394 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:58:51.394 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:58:51.395 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:58:51.395 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:58:51.395 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:58:51.395 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:58:51.395 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:58:51.395 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:58:51.395 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:58:51.395 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.395 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:58:51.395 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T03:58:51.395 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:58:51.395 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:58:51.396 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:58:51.396 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:58:51.396 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:58:51.396 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:58:51.396 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:58:51.396 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:58:51.396 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:58:51.396 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.396 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.396 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T03:58:51.396 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T03:58:51.396 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T03:58:51.397 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T03:58:51.397 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T03:58:51.397 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T03:58:51.397 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T03:58:51.397 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T03:58:51.397 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T03:58:51.397 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:58:51.397 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T03:58:51.397 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T03:58:51.397 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T03:58:51.397 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T03:58:51.397 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T03:58:51.397 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T03:58:51.398 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T03:58:51.398 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:58:51.398 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:58:51.398 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:58:51.398 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:58:51.398 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:58:51.398 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:58:51.398 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:58:51.398 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:58:51.398 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:58:51.398 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.398 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T03:58:51.399 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.399 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T03:58:51.399 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.399 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T03:58:51.399 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T03:58:51.399 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:58:51.399 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:58:51.399 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.399 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:58:51.399 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T03:58:51.399 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.399 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.400 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.400 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:58:51.400 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T03:58:51.400 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.400 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.400 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.400 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.400 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T03:58:51.400 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T03:58:51.400 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:58:51.400 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:58:51.401 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:58:51.401 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.401 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.401 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T03:58:51.401 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T03:58:51.401 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:58:51.401 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:58:51.401 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.402 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:58:51.402 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T03:58:51.402 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.402 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.402 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.402 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:58:51.402 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T03:58:51.402 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.402 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.403 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.403 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.403 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T03:58:51.403 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T03:58:51.403 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:58:51.403 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:58:51.403 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:58:51.403 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.403 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.403 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T03:58:51.403 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T03:58:51.403 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:58:51.403 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:58:51.403 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.404 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:58:51.404 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T03:58:51.404 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.404 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.404 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.404 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:58:51.404 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T03:58:51.404 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:58:51.404 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:58:51.405 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 3 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 6 2024-09-15T03:58:51.406 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.407 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:58:51.407 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": [ 2024-09-15T03:58:51.407 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:58:51.407 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:58:51.407 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:58:51.407 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T03:58:51.407 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:58:51.407 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:58:51.407 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:58:51.407 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T03:58:51.407 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T03:58:51.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78fea17640 1 -- 172.21.15.110:0/3173748210 >> v2:172.21.15.177:6812/28201 conn(0x7f78e4038700 msgr2=0x7f78e403abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78fea17640 1 --2- 172.21.15.110:0/3173748210 >> v2:172.21.15.177:6812/28201 conn(0x7f78e4038700 0x7f78e403abb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f78f806ae00 tx=0x7f78f0037000 comp rx=0 tx=0).stop 2024-09-15T03:58:51.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78fea17640 10 monclient: shutdown 2024-09-15T03:58:51.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78fea17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:51.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78fea17640 1 -- 172.21.15.110:0/3173748210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78f8134290 msgr2=0x7f78f8141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78fea17640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78f8134290 0x7f78f8141730 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f78ec00c5e0 tx=0x7f78ec004960 comp rx=0 tx=0).stop 2024-09-15T03:58:51.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78fcfb4640 1 -- 172.21.15.110:0/3173748210 reap_dead start 2024-09-15T03:58:51.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78fea17640 1 -- 172.21.15.110:0/3173748210 shutdown_connections 2024-09-15T03:58:51.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78fea17640 1 --2- 172.21.15.110:0/3173748210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78f8134290 0x7f78f8141730 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.379+0000 7f78fea17640 1 -- 172.21.15.110:0/3173748210 >> 172.21.15.110:0/3173748210 conn(0x7f78f809e570 msgr2=0x7f78f8136670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:51.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.380+0000 7f78fea17640 1 -- 172.21.15.110:0/3173748210 shutdown_connections 2024-09-15T03:58:51.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.380+0000 7f78fea17640 1 -- 172.21.15.110:0/3173748210 wait complete. 2024-09-15T03:58:51.410 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:58:51.410 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:58:51.410 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:58:51.410 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:58:51.410 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:58:51.410 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:58:51.410 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:58:51.421 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:58:51.421 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:58:51.421 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:58:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.554+0000 7fad56b14640 10 monclient: get_monmap_and_config 2024-09-15T03:58:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.554+0000 7fad56b14640 10 monclient: build_initial_monmap 2024-09-15T03:58:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.554+0000 7fad56b14640 10 monclient: monmap: 2024-09-15T03:58:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:51.555035+0000 2024-09-15T03:58:51.555 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:51.555035+0000 2024-09-15T03:58:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:51.556 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:51.557 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:51.557 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.554+0000 7fad56b14640 1 Processor -- start 2024-09-15T03:58:51.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.554+0000 7fad56b14640 1 -- start start 2024-09-15T03:58:51.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.554+0000 7fad56b14640 10 monclient: init 2024-09-15T03:58:51.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad56b14640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:51.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad56b14640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-15T03:58:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad56b14640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fad500a1a60 0x7fad5009d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad56b14640 10 monclient(hunting): picked mon.noname-a con 0x7fad500a1a60 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad56b14640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fad5009f870 0x7fad5009fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad56b14640 10 monclient(hunting): picked mon.noname-c con 0x7fad5009f870 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad56b14640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad500a01b0 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad56b14640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad500a1260 con 0x7fad500a01b0 2024-09-15T03:58:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad56b14640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad56b14640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad5009c870 con 0x7fad500a1a60 2024-09-15T03:58:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad56b14640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad56b14640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad500a0c60 con 0x7fad5009f870 2024-09-15T03:58:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad56b14640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad56b14640 10 monclient(hunting): authenticate will time out at 755.199351s 2024-09-15T03:58:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad550b1640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fad500a01b0 legacy=0x7fad50131640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad548b0640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7fad500a1a60 msgr2=0x7fad5009d4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad548b0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fad500a1a60 0x7fad5009d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad4ffff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fad5009f870 0x7fad5009fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad4ffff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fad5009f870 0x7fad5009fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.110:42004/0 (socket says 172.21.15.110:42004) 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad4ffff640 1 -- 172.21.15.110:0/3591484854 learned_addr learned my addr 172.21.15.110:0/3591484854 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad4ffff640 10 monclient(hunting): get_auth_request con 0x7fad5009f870 auth_method 0 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.555+0000 7fad4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.556+0000 7fad4ffff640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.556+0000 7fad4ffff640 1 -- 172.21.15.110:0/3591484854 >> v2:172.21.15.110:3300/0 conn(0x7fad500a1a60 msgr2=0x7fad5009d4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:58:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.556+0000 7fad4ffff640 1 --2- 172.21.15.110:0/3591484854 >> v2:172.21.15.110:3300/0 conn(0x7fad500a1a60 0x7fad5009d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.556+0000 7fad4ffff640 1 -- 172.21.15.110:0/3591484854 >> v1:172.21.15.110:6789/0 conn(0x7fad500a01b0 legacy=0x7fad50131640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:58:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.556+0000 7fad4ffff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.556+0000 7fad4ffff640 1 monclient: found mon.noname-c 2024-09-15T03:58:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.556+0000 7fad4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.556+0000 7fad4ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:58:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.556+0000 7fad4ffff640 1 -- 172.21.15.110:0/3591484854 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5009cb10 con 0x7fad5009f870 2024-09-15T03:58:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.558+0000 7fad4ffff640 1 -- 172.21.15.110:0/3591484854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5009f870 msgr2=0x7fad5009fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.558+0000 7fad4ffff640 1 --2- 172.21.15.110:0/3591484854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5009f870 0x7fad5009fc70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fad500a20d0 tx=0x7fad440091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d38e8a6d180d3400 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad4f7fe640 1 -- 172.21.15.110:0/3591484854 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44032440 con 0x7fad5009f870 2024-09-15T03:58:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad4f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad4f7fe640 10 monclient: dump: 2024-09-15T03:58:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.571 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad4f7fe640 10 monclient: _finish_auth 0 2024-09-15T03:58:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad4f7fe640 1 -- 172.21.15.110:0/3591484854 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad440325a0 con 0x7fad5009f870 2024-09-15T03:58:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad56b14640 5 monclient: authenticate success, global_id 4336 2024-09-15T03:58:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad56b14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad56b14640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad56b14640 4 set_mon_vals no callback set 2024-09-15T03:58:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad4f7fe640 1 -- 172.21.15.110:0/3591484854 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad44032940 con 0x7fad5009f870 2024-09-15T03:58:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad4f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad4f7fe640 10 monclient: dump: 2024-09-15T03:58:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.572 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad56b14640 10 monclient: shutdown 2024-09-15T03:58:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad56b14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad56b14640 1 -- 172.21.15.110:0/3591484854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5009f870 msgr2=0x7fad5009fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad56b14640 1 --2- 172.21.15.110:0/3591484854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5009f870 0x7fad5009fc70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fad500a20d0 tx=0x7fad440091d0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad56b14640 1 -- 172.21.15.110:0/3591484854 shutdown_connections 2024-09-15T03:58:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad56b14640 1 --2- 172.21.15.110:0/3591484854 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad5009f870 0x7fad5009fc70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad56b14640 1 --2- 172.21.15.110:0/3591484854 >> v2:172.21.15.110:3300/0 conn(0x7fad500a1a60 0x7fad5009d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad56b14640 1 -- 172.21.15.110:0/3591484854 >> 172.21.15.110:0/3591484854 conn(0x7fad50063270 msgr2=0x7fad50063670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad56b14640 1 -- 172.21.15.110:0/3591484854 shutdown_connections 2024-09-15T03:58:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.559+0000 7fad56b14640 1 -- 172.21.15.110:0/3591484854 wait complete. 2024-09-15T03:58:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.560+0000 7fad56b14640 10 monclient: build_initial_monmap 2024-09-15T03:58:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.560+0000 7fad56b14640 10 monclient: monmap: 2024-09-15T03:58:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:51.574 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.560+0000 7fad56b14640 1 Processor -- start 2024-09-15T03:58:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.560+0000 7fad56b14640 1 -- start start 2024-09-15T03:58:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.560+0000 7fad56b14640 10 monclient: init 2024-09-15T03:58:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad5009f870 0x7fad50141200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 10 monclient(hunting): picked mon.noname-c con 0x7fad5009f870 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad500a01b0 0x7fad50141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 10 monclient(hunting): picked mon.noname-b con 0x7fad500a01b0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad500a1a60 0x7fad5011e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 10 monclient(hunting): picked mon.noname-a con 0x7fad500a1a60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad501314d0 con 0x7fad500a1a60 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad500a14c0 con 0x7fad500a01b0 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad501317e0 con 0x7fad5009f870 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad56b14640 10 monclient(hunting): authenticate will time out at 755.205110s 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad550b1640 1 -- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad500a1a60 msgr2=0x7fad5011e2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:58:51.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad550b1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad500a1a60 0x7fad5011e2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:58:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad5009f870 0x7fad50141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad5009f870 0x7fad50141200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43658/0 (socket says 172.21.15.110:43658) 2024-09-15T03:58:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 1 -- 172.21.15.110:0/4243459078 learned_addr learned my addr 172.21.15.110:0/4243459078 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 10 monclient(hunting): get_auth_request con 0x7fad5009f870 auth_method 0 2024-09-15T03:58:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad4ffff640 1 --2- 172.21.15.110:0/4243459078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad500a01b0 0x7fad50141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad4ffff640 10 monclient(hunting): get_auth_request con 0x7fad500a01b0 auth_method 0 2024-09-15T03:58:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 10 monclient(hunting): handle_auth_done global_id 4397 payload 824 2024-09-15T03:58:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 1 -- 172.21.15.110:0/4243459078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad500a01b0 msgr2=0x7fad50141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 1 --2- 172.21.15.110:0/4243459078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad500a01b0 0x7fad50141740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 1 -- 172.21.15.110:0/4243459078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad500a1a60 msgr2=0x7fad5011e2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:58:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 1 --2- 172.21.15.110:0/4243459078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad500a1a60 0x7fad5011e2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 10 monclient: _finish_hunting 0 2024-09-15T03:58:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 1 monclient: found mon.noname-c 2024-09-15T03:58:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.561+0000 7fad548b0640 1 -- 172.21.15.110:0/4243459078 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad44009450 con 0x7fad5009f870 2024-09-15T03:58:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad548b0640 1 -- 172.21.15.110:0/4243459078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad5009f870 msgr2=0x7fad50141200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad548b0640 1 --2- 172.21.15.110:0/4243459078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad5009f870 0x7fad50141200 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fad4800bbb0 tx=0x7fad480097f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad4cff9640 1 -- 172.21.15.110:0/4243459078 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4800cc20 con 0x7fad5009f870 2024-09-15T03:58:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad4cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad4cff9640 10 monclient: dump: 2024-09-15T03:58:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.579 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad4cff9640 10 monclient: _finish_auth 0 2024-09-15T03:58:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad4cff9640 1 -- 172.21.15.110:0/4243459078 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4800cd80 con 0x7fad5009f870 2024-09-15T03:58:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad56b14640 5 monclient: authenticate success, global_id 4397 2024-09-15T03:58:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad56b14640 10 monclient: _renew_subs 2024-09-15T03:58:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad56b14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad56b14640 1 -- 172.21.15.110:0/4243459078 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5011fcd0 con 0x7fad5009f870 2024-09-15T03:58:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad4cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad4cff9640 1 -- 172.21.15.110:0/4243459078 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad48005340 con 0x7fad5009f870 2024-09-15T03:58:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad4cff9640 10 monclient: dump: 2024-09-15T03:58:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:51.581 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad4effd640 4 set_mon_vals no callback set 2024-09-15T03:58:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad56b14640 10 monclient: _renew_subs 2024-09-15T03:58:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad56b14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.563+0000 7fad56b14640 1 -- 172.21.15.110:0/4243459078 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad501208a0 con 0x7fad5009f870 2024-09-15T03:58:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.564+0000 7fad4cff9640 1 -- 172.21.15.110:0/4243459078 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fad48012070 con 0x7fad5009f870 2024-09-15T03:58:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.564+0000 7fad56b14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.564+0000 7fad56b14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.564+0000 7fad56b14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.564+0000 7fad56b14640 1 -- 172.21.15.110:0/4243459078 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad50121f70 con 0x7fad5009f870 2024-09-15T03:58:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.564+0000 7fad4cff9640 1 --2- 172.21.15.110:0/4243459078 >> v2:172.21.15.177:6812/28201 conn(0x7fad3c0386b0 0x7fad3c03ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.564+0000 7fad4cff9640 1 -- 172.21.15.110:0/4243459078 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fad4804b810 con 0x7fad5009f870 2024-09-15T03:58:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.565+0000 7fad4ffff640 1 --2- 172.21.15.110:0/4243459078 >> v2:172.21.15.177:6812/28201 conn(0x7fad3c0386b0 0x7fad3c03ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.565+0000 7fad4ffff640 10 monclient: get_auth_request con 0x7fad3c0386b0 auth_method 0 2024-09-15T03:58:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.567+0000 7fad4ffff640 1 --2- 172.21.15.110:0/4243459078 >> v2:172.21.15.177:6812/28201 conn(0x7fad3c0386b0 0x7fad3c03ab60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fad500a2900 tx=0x7fad44032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:51.569+0000 7fad548b0640 1 --2- 172.21.15.110:0/4243459078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad5009f870 0x7fad50141200 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fad4800bbb0 tx=0x7fad480097f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7fad50121f70 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 2024-09-15T03:58:52.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.397+0000 7fad4cff9640 1 -- 172.21.15.110:0/4243459078 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fad50121f70 con 0x7fad5009f870 2024-09-15T03:58:52.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.397+0000 7fad4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:52.398 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.397+0000 7fad4cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:52.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.613+0000 7fad56b14640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:58:52.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.613+0000 7fad56b14640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:58:52.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.613+0000 7fad56b14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:52.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.613+0000 7fad56b14640 1 -- 172.21.15.110:0/4243459078 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fad50121f70 con 0x7fad5009f870 2024-09-15T03:58:52.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.614+0000 7fad4cff9640 1 -- 172.21.15.110:0/4243459078 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fad50121f70 con 0x7fad5009f870 2024-09-15T03:58:52.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.614+0000 7fad4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:58:52.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.614+0000 7fad4cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:52.615 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:58:52.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.615+0000 7fad56b14640 1 -- 172.21.15.110:0/4243459078 >> v2:172.21.15.177:6812/28201 conn(0x7fad3c0386b0 msgr2=0x7fad3c03ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:52.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.615+0000 7fad56b14640 1 --2- 172.21.15.110:0/4243459078 >> v2:172.21.15.177:6812/28201 conn(0x7fad3c0386b0 0x7fad3c03ab60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fad500a2900 tx=0x7fad44032000 comp rx=0 tx=0).stop 2024-09-15T03:58:52.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.616+0000 7fad56b14640 10 monclient: shutdown 2024-09-15T03:58:52.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.616+0000 7fad56b14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:52.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.616+0000 7fad56b14640 1 -- 172.21.15.110:0/4243459078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad5009f870 msgr2=0x7fad50141200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:52.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.616+0000 7fad56b14640 1 --2- 172.21.15.110:0/4243459078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad5009f870 0x7fad50141200 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fad4800bbb0 tx=0x7fad480097f0 comp rx=0 tx=0).stop 2024-09-15T03:58:52.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.616+0000 7fad56b14640 1 -- 172.21.15.110:0/4243459078 shutdown_connections 2024-09-15T03:58:52.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.616+0000 7fad56b14640 1 --2- 172.21.15.110:0/4243459078 >> v2:172.21.15.177:6812/28201 conn(0x7fad3c0386b0 0x7fad3c03ab60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:52.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.616+0000 7fad56b14640 1 --2- 172.21.15.110:0/4243459078 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fad500a1a60 0x7fad5011e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:52.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.616+0000 7fad56b14640 1 --2- 172.21.15.110:0/4243459078 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fad500a01b0 0x7fad50141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:52.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.616+0000 7fad56b14640 1 --2- 172.21.15.110:0/4243459078 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fad5009f870 0x7fad50141200 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:52.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.616+0000 7fad56b14640 1 -- 172.21.15.110:0/4243459078 >> 172.21.15.110:0/4243459078 conn(0x7fad50063270 msgr2=0x7fad500a35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:52.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.616+0000 7fad56b14640 1 -- 172.21.15.110:0/4243459078 shutdown_connections 2024-09-15T03:58:52.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:52.616+0000 7fad56b14640 1 -- 172.21.15.110:0/4243459078 wait complete. 2024-09-15T03:58:55.628 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:58:55.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.730+0000 7f50b8f1d640 10 monclient: get_monmap_and_config 2024-09-15T03:58:55.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.730+0000 7f50b8f1d640 10 monclient: build_initial_monmap 2024-09-15T03:58:55.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.730+0000 7f50b8f1d640 10 monclient: monmap: 2024-09-15T03:58:55.732 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:55.732 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:55.732 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:55.731774+0000 2024-09-15T03:58:55.732 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:55.731774+0000 2024-09-15T03:58:55.732 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:55.732 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:55.732 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:55.732 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:55.732 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:55.732 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:55.732 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:55.732 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:55.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.731+0000 7f50b8f1d640 1 Processor -- start 2024-09-15T03:58:55.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.731+0000 7f50b8f1d640 1 -- start start 2024-09-15T03:58:55.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.731+0000 7f50b8f1d640 10 monclient: init 2024-09-15T03:58:55.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b8f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:55.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b8f1d640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-15T03:58:55.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b8f1d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f50b4131630 0x7f50b4131a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:55.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b8f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f50b4131630 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:55.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b8f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f50b4131f40 addr v1:172.21.15.110:6789/0 2024-09-15T03:58:55.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b8f1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50b406ecb0 addr v1:172.21.15.177:6789/0 2024-09-15T03:58:55.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b8f1d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b409f320 con 0x7f50b4131f40 2024-09-15T03:58:55.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b8f1d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b4068390 con 0x7f50b406ecb0 2024-09-15T03:58:55.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b8f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:55.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b8f1d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b4132310 con 0x7f50b4131630 2024-09-15T03:58:55.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b8f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:55.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b1d9b640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f50b4131f40 legacy=0x7f50b406e5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-09-15T03:58:55.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b8f1d640 10 monclient(hunting): authenticate will time out at 759.376189s 2024-09-15T03:58:55.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f50b4131630 0x7f50b4131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:55.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b2d9d640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f50b406ecb0 0x7f50b4138970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.110:41686/0 (socket says 172.21.15.110:41686) 2024-09-15T03:58:55.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f50b4131630 0x7f50b4131a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.110:42022/0 (socket says 172.21.15.110:42022) 2024-09-15T03:58:55.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b2d9d640 1 -- 172.21.15.110:0/1947304954 learned_addr learned my addr 172.21.15.110:0/1947304954 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:58:55.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 10 monclient(hunting): get_auth_request con 0x7f50b4131630 auth_method 0 2024-09-15T03:58:55.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:55.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:55.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:55.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:55.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:55.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:55.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 10 monclient(hunting): handle_auth_done global_id 4342 payload 274 2024-09-15T03:58:55.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 1 -- 172.21.15.110:0/1947304954 >> v1:172.21.15.177:6789/0 conn(0x7f50b406ecb0 legacy=0x7f50b4138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:55.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 1 -- 172.21.15.110:0/1947304954 >> v1:172.21.15.110:6789/0 conn(0x7f50b4131f40 legacy=0x7f50b406e5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:58:55.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 10 monclient: _finish_hunting 0 2024-09-15T03:58:55.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 1 monclient: found mon.noname-c 2024-09-15T03:58:55.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:55.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:58:55.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.732+0000 7f50b259c640 1 -- 172.21.15.110:0/1947304954 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b4069570 con 0x7f50b4131630 2024-09-15T03:58:55.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.735+0000 7f50b259c640 1 -- 172.21.15.110:0/1947304954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50b4131630 msgr2=0x7f50b4131a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:55.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.735+0000 7f50b259c640 1 --2- 172.21.15.110:0/1947304954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50b4131630 0x7f50b4131a00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009e60 tx=0x7f50ac009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f80a283ee5af54f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:55.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.735+0000 7f50b159a640 1 -- 172.21.15.110:0/1947304954 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac007840 con 0x7f50b4131630 2024-09-15T03:58:55.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.735+0000 7f50b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:55.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:55.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b159a640 10 monclient: dump: 2024-09-15T03:58:55.743 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:55.743 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:55.743 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:55.743 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:55.743 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:55.743 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:55.743 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:55.743 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:55.744 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:55.744 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:55.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b159a640 10 monclient: _finish_auth 0 2024-09-15T03:58:55.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:55.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b159a640 1 -- 172.21.15.110:0/1947304954 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50ac02d050 con 0x7f50b4131630 2024-09-15T03:58:55.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b8f1d640 5 monclient: authenticate success, global_id 4342 2024-09-15T03:58:55.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b8f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:55.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:55.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b8f1d640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:55.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b8f1d640 4 set_mon_vals no callback set 2024-09-15T03:58:55.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b159a640 1 -- 172.21.15.110:0/1947304954 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac007bd0 con 0x7f50b4131630 2024-09-15T03:58:55.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:55.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:58:55.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b159a640 10 monclient: dump: 2024-09-15T03:58:55.745 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:55.745 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:55.745 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:55.745 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:55.745 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:55.745 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:55.745 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:55.745 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:55.745 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:55.745 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:55.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b8f1d640 10 monclient: shutdown 2024-09-15T03:58:55.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b8f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:55.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1947304954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50b4131630 msgr2=0x7f50b4131a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:55.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b8f1d640 1 --2- 172.21.15.110:0/1947304954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50b4131630 0x7f50b4131a00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009e60 tx=0x7f50ac009780 comp rx=0 tx=0).stop 2024-09-15T03:58:55.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1947304954 shutdown_connections 2024-09-15T03:58:55.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b8f1d640 1 --2- 172.21.15.110:0/1947304954 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50b4131630 0x7f50b4131a00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:55.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1947304954 >> 172.21.15.110:0/1947304954 conn(0x7f50b40639f0 msgr2=0x7f50b40a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:55.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1947304954 shutdown_connections 2024-09-15T03:58:55.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.736+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1947304954 wait complete. 2024-09-15T03:58:55.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 10 monclient: build_initial_monmap 2024-09-15T03:58:55.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 10 monclient: monmap: 2024-09-15T03:58:55.746 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:55.746 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:55.746 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:55.746 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:55.746 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:55.746 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 1 Processor -- start 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 1 -- start start 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 10 monclient: init 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50b406ecb0 0x7f50b4137e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f50b406ecb0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50b4131630 0x7f50b4133b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f50b4131630 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b4131f40 0x7f50b4134070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:55.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f50b4131f40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4131030 con 0x7f50b406ecb0 2024-09-15T03:58:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4134fa0 con 0x7f50b4131630 2024-09-15T03:58:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b4135710 con 0x7f50b4131f40 2024-09-15T03:58:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b8f1d640 10 monclient(hunting): authenticate will time out at 759.381985s 2024-09-15T03:58:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.737+0000 7f50b259c640 1 -- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50b406ecb0 msgr2=0x7f50b4137e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:58:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b259c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50b406ecb0 0x7f50b4137e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:58:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b4131f40 0x7f50b4134070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b4131f40 0x7f50b4134070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43660/0 (socket says 172.21.15.110:43660) 2024-09-15T03:58:55.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 1 -- 172.21.15.110:0/1862933075 learned_addr learned my addr 172.21.15.110:0/1862933075 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:55.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 10 monclient(hunting): get_auth_request con 0x7f50b4131f40 auth_method 0 2024-09-15T03:58:55.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:55.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:55.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:55.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:55.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:55.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:55.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b1d9b640 1 --2- 172.21.15.110:0/1862933075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50b4131630 0x7f50b4133b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:55.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b1d9b640 10 monclient(hunting): get_auth_request con 0x7f50b4131630 auth_method 0 2024-09-15T03:58:55.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:55.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:55.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:55.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:55.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:55.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:55.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 10 monclient(hunting): handle_auth_done global_id 4400 payload 824 2024-09-15T03:58:55.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 1 -- 172.21.15.110:0/1862933075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50b4131630 msgr2=0x7f50b4133b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:55.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 1 --2- 172.21.15.110:0/1862933075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50b4131630 0x7f50b4133b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:55.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 1 -- 172.21.15.110:0/1862933075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50b406ecb0 msgr2=0x7f50b4137e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:58:55.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 1 --2- 172.21.15.110:0/1862933075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50b406ecb0 0x7f50b4137e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:55.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 10 monclient: _finish_hunting 0 2024-09-15T03:58:55.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 1 monclient: found mon.noname-c 2024-09-15T03:58:55.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:55.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:55.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.738+0000 7f50b2d9d640 1 -- 172.21.15.110:0/1862933075 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac009a20 con 0x7f50b4131f40 2024-09-15T03:58:55.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50b2d9d640 1 -- 172.21.15.110:0/1862933075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b4131f40 msgr2=0x7f50b4134070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:55.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50b2d9d640 1 --2- 172.21.15.110:0/1862933075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b4131f40 0x7f50b4134070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f50a400c510 tx=0x7f50a4004900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:55.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50aa7fc640 1 -- 172.21.15.110:0/1862933075 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4005cb0 con 0x7f50b4131f40 2024-09-15T03:58:55.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:55.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50aa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:55.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50aa7fc640 10 monclient: dump: 2024-09-15T03:58:55.751 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:55.751 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:55.751 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:55.751 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:55.751 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:55.751 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:55.751 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:55.751 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:55.751 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:55.751 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:55.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50aa7fc640 10 monclient: _finish_auth 0 2024-09-15T03:58:55.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:55.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50b8f1d640 5 monclient: authenticate success, global_id 4400 2024-09-15T03:58:55.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50aa7fc640 1 -- 172.21.15.110:0/1862933075 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a4005e30 con 0x7f50b4131f40 2024-09-15T03:58:55.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:55.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50aa7fc640 1 -- 172.21.15.110:0/1862933075 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4007320 con 0x7f50b4131f40 2024-09-15T03:58:55.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:55.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:55.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50aa7fc640 10 monclient: dump: 2024-09-15T03:58:55.752 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:55.752 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:55.752 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:55.752 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:55.752 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:55.752 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:55.753 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:55.753 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:55.753 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:55.753 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:55.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50b8f1d640 10 monclient: _renew_subs 2024-09-15T03:58:55.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50b8f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:55.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50b0d99640 4 set_mon_vals no callback set 2024-09-15T03:58:55.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1862933075 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b41371d0 con 0x7f50b4131f40 2024-09-15T03:58:55.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50b8f1d640 10 monclient: _renew_subs 2024-09-15T03:58:55.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50b8f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:55.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.740+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1862933075 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b41362e0 con 0x7f50b4131f40 2024-09-15T03:58:55.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.741+0000 7f50aa7fc640 1 -- 172.21.15.110:0/1862933075 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f50a401b3b0 con 0x7f50b4131f40 2024-09-15T03:58:55.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.741+0000 7f50b8f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:55.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.741+0000 7f50b8f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:55.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.741+0000 7f50b8f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:55.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.741+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1862933075 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b41376b0 con 0x7f50b4131f40 2024-09-15T03:58:55.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.741+0000 7f50aa7fc640 1 --2- 172.21.15.110:0/1862933075 >> v2:172.21.15.177:6812/28201 conn(0x7f50940386b0 0x7f509403ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:55.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.742+0000 7f50aa7fc640 1 -- 172.21.15.110:0/1862933075 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f50a404ac50 con 0x7f50b4131f40 2024-09-15T03:58:55.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.742+0000 7f50b259c640 1 --2- 172.21.15.110:0/1862933075 >> v2:172.21.15.177:6812/28201 conn(0x7f50940386b0 0x7f509403ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:55.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.742+0000 7f50b259c640 10 monclient: get_auth_request con 0x7f50940386b0 auth_method 0 2024-09-15T03:58:55.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.745+0000 7f50b259c640 1 --2- 172.21.15.110:0/1862933075 >> v2:172.21.15.177:6812/28201 conn(0x7f50940386b0 0x7f509403ab60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f50b4069d10 tx=0x7f50ac037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:55.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.746+0000 7f50aa7fc640 1 -- 172.21.15.110:0/1862933075 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f50a4007750 con 0x7f50b4131f40 2024-09-15T03:58:55.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.746+0000 7f50aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:55.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.746+0000 7f50aa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:55.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.961+0000 7f50b8f1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:58:55.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.961+0000 7f50b8f1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:58:55.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.961+0000 7f50b8f1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:55.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.961+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1862933075 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f50b4070950 con 0x7f50b4131f40 2024-09-15T03:58:55.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.961+0000 7f50aa7fc640 1 -- 172.21.15.110:0/1862933075 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f50a4017020 con 0x7f50b4131f40 2024-09-15T03:58:55.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.961+0000 7f50aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:58:55.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.961+0000 7f50aa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:55.963 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:58:55.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.963+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1862933075 >> v2:172.21.15.177:6812/28201 conn(0x7f50940386b0 msgr2=0x7f509403ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:55.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.963+0000 7f50b8f1d640 1 --2- 172.21.15.110:0/1862933075 >> v2:172.21.15.177:6812/28201 conn(0x7f50940386b0 0x7f509403ab60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f50b4069d10 tx=0x7f50ac037000 comp rx=0 tx=0).stop 2024-09-15T03:58:55.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.963+0000 7f50b8f1d640 10 monclient: shutdown 2024-09-15T03:58:55.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.963+0000 7f50b8f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:55.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.963+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1862933075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b4131f40 msgr2=0x7f50b4134070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:55.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.963+0000 7f50b8f1d640 1 --2- 172.21.15.110:0/1862933075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b4131f40 0x7f50b4134070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f50a400c510 tx=0x7f50a4004900 comp rx=0 tx=0).stop 2024-09-15T03:58:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.963+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1862933075 shutdown_connections 2024-09-15T03:58:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.963+0000 7f50b8f1d640 1 --2- 172.21.15.110:0/1862933075 >> v2:172.21.15.177:6812/28201 conn(0x7f50940386b0 0x7f509403ab60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.963+0000 7f50b8f1d640 1 --2- 172.21.15.110:0/1862933075 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f50b4131f40 0x7f50b4134070 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.963+0000 7f50b8f1d640 1 --2- 172.21.15.110:0/1862933075 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50b4131630 0x7f50b4133b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.963+0000 7f50b8f1d640 1 --2- 172.21.15.110:0/1862933075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f50b406ecb0 0x7f50b4137e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.963+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1862933075 >> 172.21.15.110:0/1862933075 conn(0x7f50b40639f0 msgr2=0x7f50b4139700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.963+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1862933075 shutdown_connections 2024-09-15T03:58:55.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:55.963+0000 7f50b8f1d640 1 -- 172.21.15.110:0/1862933075 wait complete. 2024-09-15T03:58:58.975 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:58:59.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.079+0000 7fe2030b8640 10 monclient: get_monmap_and_config 2024-09-15T03:58:59.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.079+0000 7fe2030b8640 10 monclient: build_initial_monmap 2024-09-15T03:58:59.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.079+0000 7fe2030b8640 10 monclient: monmap: 2024-09-15T03:58:59.080 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:59.080 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:59.080 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:59.080158+0000 2024-09-15T03:58:59.080 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:59.080158+0000 2024-09-15T03:58:59.080 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:59.080 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:59.081 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:58:59.081 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:58:59.081 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:58:59.081 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:58:59.081 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:58:59.081 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:59.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.079+0000 7fe2030b8640 1 Processor -- start 2024-09-15T03:58:59.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 1 -- start start 2024-09-15T03:58:59.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 10 monclient: init 2024-09-15T03:58:59.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:59.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-15T03:58:59.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe1fc0a1a60 0x7fe1fc09d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:59.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 10 monclient(hunting): picked mon.noname-a con 0x7fe1fc0a1a60 addr v2:172.21.15.110:3300/0 2024-09-15T03:58:59.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 0x7fe1fc09fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:59.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 10 monclient(hunting): picked mon.noname-b con 0x7fe1fc09f870 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:59.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe1fc0a0180 0x7fe1fc131640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:59.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 10 monclient(hunting): picked mon.noname-c con 0x7fe1fc0a0180 addr v2:172.21.15.177:3300/0 2024-09-15T03:58:59.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:59.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1fc0a1260 con 0x7fe1fc0a1a60 2024-09-15T03:58:59.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:59.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1fc09c870 con 0x7fe1fc0a0180 2024-09-15T03:58:59.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:59.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc0a0c60 con 0x7fe1fc09f870 2024-09-15T03:58:59.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:59.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe2030b8640 10 monclient(hunting): authenticate will time out at 762.724629s 2024-09-15T03:58:59.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe200e54640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7fe1fc0a1a60 msgr2=0x7fe1fc09d4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:58:59.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe200e54640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe1fc0a1a60 0x7fe1fc09d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:58:59.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe1fbfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 0x7fe1fc09fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:59.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe1fbfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 0x7fe1fc09fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43664/0 (socket says 172.21.15.110:43664) 2024-09-15T03:58:59.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe1fbfff640 1 -- 172.21.15.110:0/1604564995 learned_addr learned my addr 172.21.15.110:0/1604564995 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:59.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe1fbfff640 10 monclient(hunting): get_auth_request con 0x7fe1fc09f870 auth_method 0 2024-09-15T03:58:59.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe1fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:59.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe1fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:59.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe1fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:59.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe1fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:59.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe1fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:59.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.080+0000 7fe1fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:59.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.081+0000 7fe1fbfff640 10 monclient(hunting): handle_auth_done global_id 4403 payload 274 2024-09-15T03:58:59.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.081+0000 7fe1fbfff640 1 -- 172.21.15.110:0/1604564995 >> v2:172.21.15.177:3300/0 conn(0x7fe1fc0a0180 msgr2=0x7fe1fc131640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:58:59.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.081+0000 7fe1fbfff640 1 --2- 172.21.15.110:0/1604564995 >> v2:172.21.15.177:3300/0 conn(0x7fe1fc0a0180 0x7fe1fc131640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:59.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.081+0000 7fe1fbfff640 1 -- 172.21.15.110:0/1604564995 >> v2:172.21.15.110:3300/0 conn(0x7fe1fc0a1a60 msgr2=0x7fe1fc09d4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:58:59.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.081+0000 7fe1fbfff640 1 --2- 172.21.15.110:0/1604564995 >> v2:172.21.15.110:3300/0 conn(0x7fe1fc0a1a60 0x7fe1fc09d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:59.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.081+0000 7fe1fbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:58:59.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.081+0000 7fe1fbfff640 1 monclient: found mon.noname-b 2024-09-15T03:58:59.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.081+0000 7fe1fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:59.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.081+0000 7fe1fbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T03:58:59.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.081+0000 7fe1fbfff640 1 -- 172.21.15.110:0/1604564995 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1fc09cb10 con 0x7fe1fc09f870 2024-09-15T03:58:59.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.083+0000 7fe1fbfff640 1 -- 172.21.15.110:0/1604564995 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 msgr2=0x7fe1fc09fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:59.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.083+0000 7fe1fbfff640 1 --2- 172.21.15.110:0/1604564995 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 0x7fe1fc09fc40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc06f730 tx=0x7fe1ec009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc30c236597cd57b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:59.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe1fb7fe640 1 -- 172.21.15.110:0/1604564995 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec007610 con 0x7fe1fc09f870 2024-09-15T03:58:59.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe1fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:59.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe1fb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:58:59.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe1fb7fe640 10 monclient: dump: 2024-09-15T03:58:59.093 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:59.093 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:59.093 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:59.093 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:59.093 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:59.093 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:59.093 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:59.094 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:59.094 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:59.094 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:59.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe1fb7fe640 10 monclient: _finish_auth 0 2024-09-15T03:58:59.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe1fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:59.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe1fb7fe640 1 -- 172.21.15.110:0/1604564995 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ec02d050 con 0x7fe1fc09f870 2024-09-15T03:58:59.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 5 monclient: authenticate success, global_id 4403 2024-09-15T03:58:59.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:58:59.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe1fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:59.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 10 monclient: get_monmap_and_config success 2024-09-15T03:58:59.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 4 set_mon_vals no callback set 2024-09-15T03:58:59.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe1fb7fe640 1 -- 172.21.15.110:0/1604564995 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec0079a0 con 0x7fe1fc09f870 2024-09-15T03:58:59.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe1fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:59.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe1fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:59.095 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe1fb7fe640 10 monclient: dump: 2024-09-15T03:58:59.095 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:59.095 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:59.095 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:59.095 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:59.095 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:59.095 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:59.095 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:59.095 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:59.095 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:59.095 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 10 monclient: shutdown 2024-09-15T03:58:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 1 -- 172.21.15.110:0/1604564995 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 msgr2=0x7fe1fc09fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 1 --2- 172.21.15.110:0/1604564995 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 0x7fe1fc09fc40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc06f730 tx=0x7fe1ec009840 comp rx=0 tx=0).stop 2024-09-15T03:58:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 1 -- 172.21.15.110:0/1604564995 shutdown_connections 2024-09-15T03:58:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 1 --2- 172.21.15.110:0/1604564995 >> v2:172.21.15.177:3300/0 conn(0x7fe1fc0a0180 0x7fe1fc131640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 1 --2- 172.21.15.110:0/1604564995 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 0x7fe1fc09fc40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 1 --2- 172.21.15.110:0/1604564995 >> v2:172.21.15.110:3300/0 conn(0x7fe1fc0a1a60 0x7fe1fc09d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 1 -- 172.21.15.110:0/1604564995 >> 172.21.15.110:0/1604564995 conn(0x7fe1fc063270 msgr2=0x7fe1fc063670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 1 -- 172.21.15.110:0/1604564995 shutdown_connections 2024-09-15T03:58:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.084+0000 7fe2030b8640 1 -- 172.21.15.110:0/1604564995 wait complete. 2024-09-15T03:58:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.085+0000 7fe2030b8640 10 monclient: build_initial_monmap 2024-09-15T03:58:59.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.085+0000 7fe2030b8640 10 monclient: monmap: 2024-09-15T03:58:59.097 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:58:59.097 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:58:59.097 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:58:59.097 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:58:59.097 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:58:59.097 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:59.097 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:58:59.097 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:58:59.097 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:58:59.097 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:59.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.085+0000 7fe2030b8640 1 Processor -- start 2024-09-15T03:58:59.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.085+0000 7fe2030b8640 1 -- start start 2024-09-15T03:58:59.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.085+0000 7fe2030b8640 10 monclient: init 2024-09-15T03:58:59.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 10 monclient: _reopen_session rank -1 2024-09-15T03:58:59.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:58:59.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 0x7fe1fc099a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:59.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 10 monclient(hunting): picked mon.noname-c con 0x7fe1fc09f870 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:58:59.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1fc0a0180 0x7fe1fc130520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:59.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 10 monclient(hunting): picked mon.noname-b con 0x7fe1fc0a0180 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:58:59.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1fc0a1a60 0x7fe1fc11b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:59.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 10 monclient(hunting): picked mon.noname-a con 0x7fe1fc0a1a60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:58:59.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:59.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc136430 con 0x7fe1fc0a1a60 2024-09-15T03:58:59.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:59.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc11dc00 con 0x7fe1fc0a0180 2024-09-15T03:58:59.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 10 monclient(hunting): start opening mon connection 2024-09-15T03:58:59.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1fc11d330 con 0x7fe1fc09f870 2024-09-15T03:58:59.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 10 monclient(hunting): _renew_subs 2024-09-15T03:58:59.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe2030b8640 10 monclient(hunting): authenticate will time out at 762.730303s 2024-09-15T03:58:59.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 0x7fe1fc099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:59.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe201655640 1 -- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1fc0a1a60 msgr2=0x7fe1fc11b4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:58:59.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe201655640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1fc0a1a60 0x7fe1fc11b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:58:59.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 0x7fe1fc099a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:43680/0 (socket says 172.21.15.110:43680) 2024-09-15T03:58:59.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 1 -- 172.21.15.110:0/1542297716 learned_addr learned my addr 172.21.15.110:0/1542297716 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:58:59.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 10 monclient(hunting): get_auth_request con 0x7fe1fc09f870 auth_method 0 2024-09-15T03:58:59.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:58:59.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:58:59.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:58:59.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:58:59.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:58:59.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:58:59.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 10 monclient(hunting): handle_auth_done global_id 4406 payload 824 2024-09-15T03:58:59.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 1 -- 172.21.15.110:0/1542297716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1fc0a0180 msgr2=0x7fe1fc130520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:59.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 1 --2- 172.21.15.110:0/1542297716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1fc0a0180 0x7fe1fc130520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:59.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 1 -- 172.21.15.110:0/1542297716 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1fc0a1a60 msgr2=0x7fe1fc11b4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:58:59.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 1 --2- 172.21.15.110:0/1542297716 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1fc0a1a60 0x7fe1fc11b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:59.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 10 monclient: _finish_hunting 0 2024-09-15T03:58:59.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 1 monclient: found mon.noname-c 2024-09-15T03:58:59.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:58:59.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:58:59.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.086+0000 7fe200e54640 1 -- 172.21.15.110:0/1542297716 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec009ae0 con 0x7fe1fc09f870 2024-09-15T03:58:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.088+0000 7fe200e54640 1 -- 172.21.15.110:0/1542297716 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 msgr2=0x7fe1fc099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:58:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.088+0000 7fe200e54640 1 --2- 172.21.15.110:0/1542297716 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 0x7fe1fc099a50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400b600 tx=0x7fe1f40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe1f8ff9640 1 -- 172.21.15.110:0/1542297716 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4015070 con 0x7fe1fc09f870 2024-09-15T03:58:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe1f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe1f8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:58:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe1f8ff9640 10 monclient: dump: 2024-09-15T03:58:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:59.101 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:59.102 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe1f8ff9640 10 monclient: _finish_auth 0 2024-09-15T03:58:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe1f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:58:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe1f8ff9640 1 -- 172.21.15.110:0/1542297716 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f4003d90 con 0x7fe1fc09f870 2024-09-15T03:58:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe2030b8640 5 monclient: authenticate success, global_id 4406 2024-09-15T03:58:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe2030b8640 10 monclient: _renew_subs 2024-09-15T03:58:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe2030b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe2030b8640 1 -- 172.21.15.110:0/1542297716 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1fc09a5d0 con 0x7fe1fc09f870 2024-09-15T03:58:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe1f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:58:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe1f8ff9640 1 -- 172.21.15.110:0/1542297716 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4006380 con 0x7fe1fc09f870 2024-09-15T03:58:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe1f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:58:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe1f8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:58:59.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe1f8ff9640 10 monclient: dump: 2024-09-15T03:58:59.103 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:58:59.103 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:58:59.103 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:59.103 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:58:59.103 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:58:59.103 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:58:59.103 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:58:59.103 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:58:59.103 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:58:59.103 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:58:59.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe1faffd640 4 set_mon_vals no callback set 2024-09-15T03:58:59.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe2030b8640 10 monclient: _renew_subs 2024-09-15T03:58:59.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe2030b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:59.103 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe2030b8640 1 -- 172.21.15.110:0/1542297716 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1fc11d640 con 0x7fe1fc09f870 2024-09-15T03:58:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.089+0000 7fe1f8ff9640 1 -- 172.21.15.110:0/1542297716 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fe1f4010440 con 0x7fe1fc09f870 2024-09-15T03:58:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.090+0000 7fe2030b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.090+0000 7fe2030b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.090+0000 7fe2030b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.090+0000 7fe1f8ff9640 1 --2- 172.21.15.110:0/1542297716 >> v2:172.21.15.177:6812/28201 conn(0x7fe1dc0386b0 0x7fe1dc03ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:58:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.090+0000 7fe2030b8640 1 -- 172.21.15.110:0/1542297716 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1fc070950 con 0x7fe1fc09f870 2024-09-15T03:58:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.090+0000 7fe1f8ff9640 1 -- 172.21.15.110:0/1542297716 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fe1f404b2b0 con 0x7fe1fc09f870 2024-09-15T03:58:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.090+0000 7fe1fbfff640 1 --2- 172.21.15.110:0/1542297716 >> v2:172.21.15.177:6812/28201 conn(0x7fe1dc0386b0 0x7fe1dc03ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:58:59.104 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.090+0000 7fe1fbfff640 10 monclient: get_auth_request con 0x7fe1dc0386b0 auth_method 0 2024-09-15T03:58:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.093+0000 7fe1fbfff640 1 --2- 172.21.15.110:0/1542297716 >> v2:172.21.15.177:6812/28201 conn(0x7fe1dc0386b0 0x7fe1dc03ab60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc0a0a10 tx=0x7fe1ec036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:58:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.094+0000 7fe1f8ff9640 1 -- 172.21.15.110:0/1542297716 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe1f40186d0 con 0x7fe1fc09f870 2024-09-15T03:58:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.094+0000 7fe1f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:58:59.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.094+0000 7fe1f8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:58:59.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.313+0000 7fe2030b8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:58:59.314 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.313+0000 7fe2030b8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:58:59.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.313+0000 7fe2030b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:58:59.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.313+0000 7fe2030b8640 1 -- 172.21.15.110:0/1542297716 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe1fc11d830 con 0x7fe1fc09f870 2024-09-15T03:58:59.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.313+0000 7fe1f8ff9640 1 -- 172.21.15.110:0/1542297716 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fe1f4018310 con 0x7fe1fc09f870 2024-09-15T03:58:59.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.313+0000 7fe1f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:58:59.315 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.313+0000 7fe1f8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:58:59.315 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":31,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:58:59.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.315+0000 7fe2030b8640 1 -- 172.21.15.110:0/1542297716 >> v2:172.21.15.177:6812/28201 conn(0x7fe1dc0386b0 msgr2=0x7fe1dc03ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:59.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.315+0000 7fe2030b8640 1 --2- 172.21.15.110:0/1542297716 >> v2:172.21.15.177:6812/28201 conn(0x7fe1dc0386b0 0x7fe1dc03ab60 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc0a0a10 tx=0x7fe1ec036000 comp rx=0 tx=0).stop 2024-09-15T03:58:59.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.315+0000 7fe2030b8640 10 monclient: shutdown 2024-09-15T03:58:59.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.315+0000 7fe2030b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:58:59.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.315+0000 7fe2030b8640 1 -- 172.21.15.110:0/1542297716 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 msgr2=0x7fe1fc099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:58:59.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.315+0000 7fe2030b8640 1 --2- 172.21.15.110:0/1542297716 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 0x7fe1fc099a50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400b600 tx=0x7fe1f40091a0 comp rx=0 tx=0).stop 2024-09-15T03:58:59.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.315+0000 7fe2030b8640 1 -- 172.21.15.110:0/1542297716 shutdown_connections 2024-09-15T03:58:59.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.316+0000 7fe2030b8640 1 --2- 172.21.15.110:0/1542297716 >> v2:172.21.15.177:6812/28201 conn(0x7fe1dc0386b0 0x7fe1dc03ab60 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:59.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.316+0000 7fe2030b8640 1 --2- 172.21.15.110:0/1542297716 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe1fc0a1a60 0x7fe1fc11b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:59.317 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.316+0000 7fe2030b8640 1 --2- 172.21.15.110:0/1542297716 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe1fc0a0180 0x7fe1fc130520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:59.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.316+0000 7fe2030b8640 1 --2- 172.21.15.110:0/1542297716 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe1fc09f870 0x7fe1fc099a50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:58:59.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.316+0000 7fe2030b8640 1 -- 172.21.15.110:0/1542297716 >> 172.21.15.110:0/1542297716 conn(0x7fe1fc063270 msgr2=0x7fe1fc133d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:58:59.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.316+0000 7fe2030b8640 1 -- 172.21.15.110:0/1542297716 shutdown_connections 2024-09-15T03:58:59.318 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:58:59.316+0000 7fe2030b8640 1 -- 172.21.15.110:0/1542297716 wait complete. 2024-09-15T03:59:02.328 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:59:02.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.430+0000 7fe0e8d00640 10 monclient: get_monmap_and_config 2024-09-15T03:59:02.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.430+0000 7fe0e8d00640 10 monclient: build_initial_monmap 2024-09-15T03:59:02.431 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.430+0000 7fe0e8d00640 10 monclient: monmap: 2024-09-15T03:59:02.431 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:59:02.432 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:02.432 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:59:02.431652+0000 2024-09-15T03:59:02.432 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:59:02.431652+0000 2024-09-15T03:59:02.432 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:59:02.432 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:02.432 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:59:02.432 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:59:02.432 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:59:02.432 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:59:02.432 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:59:02.432 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:02.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.431+0000 7fe0e8d00640 1 Processor -- start 2024-09-15T03:59:02.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.431+0000 7fe0e8d00640 1 -- start start 2024-09-15T03:59:02.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.431+0000 7fe0e8d00640 10 monclient: init 2024-09-15T03:59:02.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.431+0000 7fe0e8d00640 10 monclient: _reopen_session rank -1 2024-09-15T03:59:02.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e8d00640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-15T03:59:02.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e8d00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:02.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e8d00640 10 monclient(hunting): picked mon.noname-c con 0x7fe0e40a1bf0 addr v2:172.21.15.177:3300/0 2024-09-15T03:59:02.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e8d00640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0e409e780 0x7fe0e409eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:02.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e8d00640 10 monclient(hunting): picked mon.noname-a con 0x7fe0e409e780 addr v2:172.21.15.110:3300/0 2024-09-15T03:59:02.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e8d00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe0e409f760 addr v1:172.21.15.110:6789/0 2024-09-15T03:59:02.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e8d00640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e409c9b0 con 0x7fe0e409f760 2024-09-15T03:59:02.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e8d00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:02.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e8d00640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e40a0c20 con 0x7fe0e409e780 2024-09-15T03:59:02.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e8d00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:02.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e8d00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e409cc90 con 0x7fe0e40a1bf0 2024-09-15T03:59:02.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e8d00640 10 monclient(hunting): _renew_subs 2024-09-15T03:59:02.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e8d00640 10 monclient(hunting): authenticate will time out at 766.076110s 2024-09-15T03:59:02.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e2d9d640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7fe0e409f760 legacy=0x7fe0e409fba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-09-15T03:59:02.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0dbfff640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7fe0e409e780 msgr2=0x7fe0e409eb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:59:02.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0dbfff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe0e409e780 0x7fe0e409eb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:59:02.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e259c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:02.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e259c640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.110:53064/0 (socket says 172.21.15.110:53064) 2024-09-15T03:59:02.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 learned_addr learned my addr 172.21.15.110:0/3292865417 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:59:02.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e259c640 10 monclient(hunting): get_auth_request con 0x7fe0e40a1bf0 auth_method 0 2024-09-15T03:59:02.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:02.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:02.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:02.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 msgr2=0x7fe0e4131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:59:02.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 msgr2=0x7fe0e4131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:59:02.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:59:02.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.432+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:59:02.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.633+0000 7fe0e2d9d640 1 -- 172.21.15.110:0/3292865417 >> v1:172.21.15.110:6789/0 conn(0x7fe0e409f760 legacy=0x7fe0e409fba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-09-15T03:59:02.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.633+0000 7fe0dbfff640 1 -- 172.21.15.110:0/3292865417 >> v2:172.21.15.110:3300/0 conn(0x7fe0e409e780 msgr2=0x7fe0e409eb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:59:02.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.633+0000 7fe0dbfff640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.110:3300/0 conn(0x7fe0e409e780 0x7fe0e409eb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:59:02.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.633+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:02.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.633+0000 7fe0e259c640 10 monclient(hunting): get_auth_request con 0x7fe0e40a1bf0 auth_method 0 2024-09-15T03:59:02.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.633+0000 7fe0e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:02.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.633+0000 7fe0e259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:02.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.633+0000 7fe0e259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:59:02.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.633+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 msgr2=0x7fe0e4131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:59:02.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.633+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 msgr2=0x7fe0e4131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:59:02.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.633+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:59:02.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:02.633+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:59:03.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.034+0000 7fe0e2d9d640 1 -- 172.21.15.110:0/3292865417 >> v1:172.21.15.110:6789/0 conn(0x7fe0e409f760 legacy=0x7fe0e409fba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-09-15T03:59:03.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.034+0000 7fe0dbfff640 1 -- 172.21.15.110:0/3292865417 >> v2:172.21.15.110:3300/0 conn(0x7fe0e409e780 msgr2=0x7fe0e409eb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:59:03.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.034+0000 7fe0dbfff640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.110:3300/0 conn(0x7fe0e409e780 0x7fe0e409eb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:59:03.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.034+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:03.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.034+0000 7fe0e259c640 10 monclient(hunting): get_auth_request con 0x7fe0e40a1bf0 auth_method 0 2024-09-15T03:59:03.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.034+0000 7fe0e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:03.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.034+0000 7fe0e259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:03.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.034+0000 7fe0e259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:59:03.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.034+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 msgr2=0x7fe0e4131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:59:03.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.034+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 msgr2=0x7fe0e4131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:59:03.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.034+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:59:03.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.034+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:59:03.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.835+0000 7fe0dbfff640 1 -- 172.21.15.110:0/3292865417 >> v2:172.21.15.110:3300/0 conn(0x7fe0e409e780 msgr2=0x7fe0e409eb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:59:03.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.835+0000 7fe0e2d9d640 1 -- 172.21.15.110:0/3292865417 >> v1:172.21.15.110:6789/0 conn(0x7fe0e409f760 legacy=0x7fe0e409fba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-09-15T03:59:03.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.835+0000 7fe0dbfff640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.110:3300/0 conn(0x7fe0e409e780 0x7fe0e409eb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:59:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.835+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.835+0000 7fe0e259c640 10 monclient(hunting): get_auth_request con 0x7fe0e40a1bf0 auth_method 0 2024-09-15T03:59:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.835+0000 7fe0e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.835+0000 7fe0e259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.835+0000 7fe0e259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:59:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.836+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 msgr2=0x7fe0e4131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:59:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.836+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 msgr2=0x7fe0e4131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:59:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.836+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:59:03.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:03.836+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T03:59:05.433 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 10 monclient(hunting): tick 2024-09-15T03:59:05.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 1 monclient(hunting): continuing hunt 2024-09-15T03:59:05.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:59:05.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 1 -- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 msgr2=0x7fe0e4131810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:59:05.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.177:3300/0 conn(0x7fe0e40a1bf0 0x7fe0e4131810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:05.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 1 -- 172.21.15.110:0/3292865417 >> v2:172.21.15.110:3300/0 conn(0x7fe0e409e780 msgr2=0x7fe0e409eb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:59:05.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 1 --2- 172.21.15.110:0/3292865417 >> v2:172.21.15.110:3300/0 conn(0x7fe0e409e780 0x7fe0e409eb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:05.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 1 -- 172.21.15.110:0/3292865417 >> v1:172.21.15.110:6789/0 conn(0x7fe0e409f760 legacy=0x7fe0e409fba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:59:05.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 10 monclient: _add_conns ranks=[2,1] 2024-09-15T03:59:05.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0c4005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:05.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 10 monclient(hunting): picked mon.noname-b con 0x7fe0c4003670 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:59:05.434 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0c40060a0 addr v1:172.21.15.177:6789/0 2024-09-15T03:59:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 1 -- 172.21.15.110:0/3292865417 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e40a1fc0 con 0x7fe0c40060a0 2024-09-15T03:59:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 1 -- 172.21.15.110:0/3292865417 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e40a0c20 con 0x7fe0c4003670 2024-09-15T03:59:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e0d99640 10 monclient(hunting): _renew_subs 2024-09-15T03:59:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0c4005a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e259c640 10 monclient(hunting): get_auth_request con 0x7fe0c4003670 auth_method 0 2024-09-15T03:59:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 msgr2=0x7fe0c4005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:59:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 msgr2=0x7fe0c4005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:59:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0c4005a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:59:05.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.432+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0c4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:59:05.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.434+0000 7fe0dbfff640 1 -- 172.21.15.110:0/3292865417 >> v1:172.21.15.177:6789/0 conn(0x7fe0c40060a0 legacy=0x7fe0c4008550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:59:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.633+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0c4005a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.633+0000 7fe0e259c640 10 monclient(hunting): get_auth_request con 0x7fe0c4003670 auth_method 0 2024-09-15T03:59:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.633+0000 7fe0e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:05.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.633+0000 7fe0e259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:05.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.633+0000 7fe0e259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:59:05.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.633+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 msgr2=0x7fe0c4005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:59:05.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.633+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 msgr2=0x7fe0c4005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:59:05.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.633+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0c4005a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:59:05.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:05.633+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0c4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T03:59:06.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.034+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0c4005a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.034+0000 7fe0e259c640 10 monclient(hunting): get_auth_request con 0x7fe0c4003670 auth_method 0 2024-09-15T03:59:06.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.034+0000 7fe0e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.035 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.034+0000 7fe0e259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.034+0000 7fe0e259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:59:06.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.034+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 msgr2=0x7fe0c4005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T03:59:06.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.034+0000 7fe0e259c640 1 -- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 msgr2=0x7fe0c4005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:59:06.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.034+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0c4005a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:59:06.036 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.034+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0c4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T03:59:06.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.113+0000 7fe0e1d9b640 1 -- 172.21.15.110:0/3292865417 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421709296 0 0) 0x7fe0e40a1fc0 con 0x7fe0c40060a0 2024-09-15T03:59:06.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.113+0000 7fe0e1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.113+0000 7fe0e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:06.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.113+0000 7fe0e1d9b640 10 monclient(hunting): my global_id is 4348 2024-09-15T03:59:06.114 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.113+0000 7fe0e1d9b640 1 -- 172.21.15.110:0/3292865417 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d00034e0 con 0x7fe0c40060a0 2024-09-15T03:59:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 1 -- 172.21.15.110:0/3292865417 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1649465567 0 0) 0x7fe0d00034e0 con 0x7fe0c40060a0 2024-09-15T03:59:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 1 -- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 msgr2=0x7fe0c4005a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:59:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0c4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient: _finish_hunting 0 2024-09-15T03:59:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 1 monclient: found mon.noname-c-legacy 2024-09-15T03:59:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:59:06.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.177:6789/0 2024-09-15T03:59:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 1 -- 172.21.15.110:0/3292865417 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e409c9b0 con 0x7fe0c40060a0 2024-09-15T03:59:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient: _finish_auth 0 2024-09-15T03:59:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:59:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 1 -- 172.21.15.110:0/3292865417 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3165669920 0 0) 0x7fe0cc004090 con 0x7fe0c40060a0 2024-09-15T03:59:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T03:59:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient: dump: 2024-09-15T03:59:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.118 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.119 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] but i'm connected to v1:172.21.15.177:6789/0, reconnecting 2024-09-15T03:59:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient: _reopen_session rank -1 2024-09-15T03:59:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 1 -- 172.21.15.110:0/3292865417 >> v1:172.21.15.177:6789/0 conn(0x7fe0c40060a0 legacy=0x7fe0c4008550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:59:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e2d9d640 1 -- 172.21.15.110:0/3292865417 reap_dead start 2024-09-15T03:59:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d00039d0 0x7fe0d0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient(hunting): picked mon.b con 0x7fe0d00039d0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:59:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0d0004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient(hunting): picked mon.c con 0x7fe0c4003670 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:59:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d0004cc0 0x7fe0d0005130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient(hunting): picked mon.a con 0x7fe0d0004cc0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:59:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 1 -- 172.21.15.110:0/3292865417 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e40a0c20 con 0x7fe0d0004cc0 2024-09-15T03:59:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 1 -- 172.21.15.110:0/3292865417 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d000bb60 con 0x7fe0d00039d0 2024-09-15T03:59:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 1 -- 172.21.15.110:0/3292865417 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d000bcf0 con 0x7fe0c4003670 2024-09-15T03:59:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient(hunting): _renew_subs 2024-09-15T03:59:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e1d9b640 10 monclient(hunting): _finish_auth 0 2024-09-15T03:59:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e2d9d640 1 -- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d0004cc0 msgr2=0x7fe0d0005130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:59:06.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e2d9d640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d0004cc0 0x7fe0d0005130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:59:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0dbfff640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0d0004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0dbfff640 10 monclient(hunting): get_auth_request con 0x7fe0c4003670 auth_method 0 2024-09-15T03:59:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0dbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:59:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d00039d0 0x7fe0d0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.114+0000 7fe0e259c640 10 monclient(hunting): get_auth_request con 0x7fe0d00039d0 auth_method 0 2024-09-15T03:59:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0e259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0e259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:59:06.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:59:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:59:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:59:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:59:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:59:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0e259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:59:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 10 monclient(hunting): handle_auth_done global_id 4348 payload 293 2024-09-15T03:59:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 1 -- 172.21.15.110:0/3292865417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d00039d0 msgr2=0x7fe0d0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:59:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d00039d0 0x7fe0d0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 1 -- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d0004cc0 msgr2=0x7fe0d0005130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:59:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d0004cc0 0x7fe0d0005130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 10 monclient: _finish_hunting 0 2024-09-15T03:59:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 1 monclient: found mon.c 2024-09-15T03:59:06.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:59:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:59:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 1 -- 172.21.15.110:0/3292865417 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d00034e0 con 0x7fe0c4003670 2024-09-15T03:59:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 10 monclient: _finish_auth 0 2024-09-15T03:59:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:59:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0e8d00640 5 monclient: authenticate success, global_id 4348 2024-09-15T03:59:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0e8d00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:59:06.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.115+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d00039d0 0x7fe0d0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T03:59:06.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0dbfff640 1 -- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 msgr2=0x7fe0d0004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:59:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0dbfff640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0d0004780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc003870 tx=0x7fe0cc007960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fcea79f3184ddf3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:59:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0e1d9b640 1 -- 172.21.15.110:0/3292865417 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc007150 con 0x7fe0c4003670 2024-09-15T03:59:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0e1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:59:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0e1d9b640 10 monclient: dump: 2024-09-15T03:59:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.132 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.133 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0e1d9b640 1 -- 172.21.15.110:0/3292865417 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0cc0072d0 con 0x7fe0c4003670 2024-09-15T03:59:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0e8d00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:59:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0e1d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:59:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0e8d00640 10 monclient: get_monmap_and_config success 2024-09-15T03:59:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0e8d00640 4 set_mon_vals no callback set 2024-09-15T03:59:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0e1d9b640 1 -- 172.21.15.110:0/3292865417 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc004800 con 0x7fe0c4003670 2024-09-15T03:59:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0e1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0e1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:59:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.118+0000 7fe0e1d9b640 10 monclient: dump: 2024-09-15T03:59:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.134 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.119+0000 7fe0e8d00640 10 monclient: shutdown 2024-09-15T03:59:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.119+0000 7fe0e8d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:59:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.119+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 msgr2=0x7fe0d0004780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.119+0000 7fe0e8d00640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0d0004780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc003870 tx=0x7fe0cc007960 comp rx=0 tx=0).stop 2024-09-15T03:59:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.119+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3292865417 shutdown_connections 2024-09-15T03:59:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.119+0000 7fe0e8d00640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d0004cc0 0x7fe0d0005130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.119+0000 7fe0e8d00640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d00039d0 0x7fe0d0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.119+0000 7fe0e8d00640 1 --2- 172.21.15.110:0/3292865417 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0d0004780 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.119+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3292865417 >> 172.21.15.110:0/3292865417 conn(0x7fe0e4062dd0 msgr2=0x7fe0e40631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:59:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.119+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3292865417 shutdown_connections 2024-09-15T03:59:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.119+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3292865417 wait complete. 2024-09-15T03:59:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.120+0000 7fe0e8d00640 10 monclient: build_initial_monmap 2024-09-15T03:59:06.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.120+0000 7fe0e8d00640 10 monclient: monmap: 2024-09-15T03:59:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:59:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:59:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:59:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:59:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:59:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:59:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:59:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:59:06.136 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.120+0000 7fe0e8d00640 1 Processor -- start 2024-09-15T03:59:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.120+0000 7fe0e8d00640 1 -- start start 2024-09-15T03:59:06.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.120+0000 7fe0e8d00640 10 monclient: init 2024-09-15T03:59:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 10 monclient: _reopen_session rank -1 2024-09-15T03:59:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:59:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0e409a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 10 monclient(hunting): picked mon.noname-c con 0x7fe0c4003670 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:59:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d00039d0 0x7fe0e41306e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 10 monclient(hunting): picked mon.noname-a con 0x7fe0d00039d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:59:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d0004cc0 0x7fe0e411b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 10 monclient(hunting): picked mon.noname-b con 0x7fe0d0004cc0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:59:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e40a30c0 con 0x7fe0d00039d0 2024-09-15T03:59:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e409acd0 con 0x7fe0d0004cc0 2024-09-15T03:59:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e4141850 con 0x7fe0c4003670 2024-09-15T03:59:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 10 monclient(hunting): _renew_subs 2024-09-15T03:59:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e8d00640 10 monclient(hunting): authenticate will time out at 769.765433s 2024-09-15T03:59:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0dbfff640 1 -- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d00039d0 msgr2=0x7fe0e41306e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:59:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0dbfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d00039d0 0x7fe0e41306e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:59:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0e409a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0e409a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:51718/0 (socket says 172.21.15.110:51718) 2024-09-15T03:59:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 1 -- 172.21.15.110:0/3819713992 learned_addr learned my addr 172.21.15.110:0/3819713992 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:59:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 10 monclient(hunting): get_auth_request con 0x7fe0c4003670 auth_method 0 2024-09-15T03:59:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:06.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e2d9d640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d0004cc0 0x7fe0e411b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e2d9d640 10 monclient(hunting): get_auth_request con 0x7fe0d0004cc0 auth_method 0 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 10 monclient(hunting): handle_auth_done global_id 4409 payload 824 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 1 -- 172.21.15.110:0/3819713992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d0004cc0 msgr2=0x7fe0e411b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d0004cc0 0x7fe0e411b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 1 -- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d00039d0 msgr2=0x7fe0e41306e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:59:06.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d00039d0 0x7fe0e41306e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.121+0000 7fe0e259c640 10 monclient: _finish_hunting 0 2024-09-15T03:59:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.122+0000 7fe0e259c640 1 monclient: found mon.noname-c 2024-09-15T03:59:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.122+0000 7fe0e259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:59:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.122+0000 7fe0e259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:59:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.122+0000 7fe0e259c640 1 -- 172.21.15.110:0/3819713992 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e411e0a0 con 0x7fe0c4003670 2024-09-15T03:59:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e259c640 1 -- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 msgr2=0x7fe0e409a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:59:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0e409a2a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4016ca0 tx=0x7fe0d4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:59:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 1 -- 172.21.15.110:0/3819713992 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d401d070 con 0x7fe0c4003670 2024-09-15T03:59:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:59:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient: dump: 2024-09-15T03:59:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.140 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.141 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient: _finish_auth 0 2024-09-15T03:59:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:59:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 1 -- 172.21.15.110:0/3819713992 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d401e040 con 0x7fe0c4003670 2024-09-15T03:59:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:59:06.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 1 -- 172.21.15.110:0/3819713992 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4016ee0 con 0x7fe0c4003670 2024-09-15T03:59:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e8d00640 5 monclient: authenticate success, global_id 4409 2024-09-15T03:59:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e8d00640 10 monclient: _renew_subs 2024-09-15T03:59:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e8d00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:59:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3819713992 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e411d800 con 0x7fe0c4003670 2024-09-15T03:59:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e159a640 4 set_mon_vals no callback set 2024-09-15T03:59:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:59:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient: dump: 2024-09-15T03:59:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.142 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e8d00640 10 monclient: _renew_subs 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e8d00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3819713992 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e4135ea0 con 0x7fe0c4003670 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e259c640 1 -- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 msgr2=0x7fe0e409a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e259c640 1 -- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 msgr2=0x7fe0e409a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0e409a2a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4016ca0 tx=0x7fe0d4008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0e409a2a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4016ca0 tx=0x7fe0d4008040 comp rx=0 tx=0).stop 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient: _reopen_session rank -1 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 1 -- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 msgr2=0x7fe0e409a2a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:59:06.143 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4003670 0x7fe0e409a2a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:59:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4009ff0 0x7fe0c400a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient(hunting): picked mon.c con 0x7fe0c4009ff0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:59:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d00039d0 0x7fe0c4008d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient(hunting): picked mon.a con 0x7fe0d00039d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:59:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d0004cc0 0x7fe0c4009750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient(hunting): picked mon.b con 0x7fe0d0004cc0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:59:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 1 -- 172.21.15.110:0/3819713992 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c4007da0 con 0x7fe0d00039d0 2024-09-15T03:59:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 1 -- 172.21.15.110:0/3819713992 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c4007ee0 con 0x7fe0d0004cc0 2024-09-15T03:59:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 1 -- 172.21.15.110:0/3819713992 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c4008020 con 0x7fe0c4009ff0 2024-09-15T03:59:06.144 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0da7fc640 10 monclient(hunting): _renew_subs 2024-09-15T03:59:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0dbfff640 1 -- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d00039d0 msgr2=0x7fe0c4008d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:59:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0dbfff640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d00039d0 0x7fe0c4008d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:59:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.124+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4009ff0 0x7fe0c400a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 10 monclient(hunting): get_auth_request con 0x7fe0c4009ff0 auth_method 0 2024-09-15T03:59:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:59:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:59:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:59:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T03:59:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e2d9d640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d0004cc0 0x7fe0c4009750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.145 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e2d9d640 10 monclient(hunting): get_auth_request con 0x7fe0d0004cc0 auth_method 0 2024-09-15T03:59:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:59:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 10 monclient(hunting): handle_auth_done global_id 4409 payload 293 2024-09-15T03:59:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 1 -- 172.21.15.110:0/3819713992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d0004cc0 msgr2=0x7fe0c4009750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d0004cc0 0x7fe0c4009750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 1 -- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d00039d0 msgr2=0x7fe0c4008d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:59:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d00039d0 0x7fe0c4008d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 10 monclient: _finish_hunting 0 2024-09-15T03:59:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 1 monclient: found mon.c 2024-09-15T03:59:06.146 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:59:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:59:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 1 -- 172.21.15.110:0/3819713992 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fe0c4008310 con 0x7fe0c4009ff0 2024-09-15T03:59:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 10 monclient: _finish_auth 0 2024-09-15T03:59:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:59:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e8d00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:59:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e8d00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:59:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e8d00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:59:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.125+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3819713992 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e411dc70 con 0x7fe0c4009ff0 2024-09-15T03:59:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.127+0000 7fe0e259c640 1 -- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4009ff0 msgr2=0x7fe0c400a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:59:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.127+0000 7fe0e259c640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4009ff0 0x7fe0c400a3c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4016c70 tx=0x7fe0d4027000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:59:06.147 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.127+0000 7fe0da7fc640 1 -- 172.21.15.110:0/3819713992 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4021ab0 con 0x7fe0c4009ff0 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.127+0000 7fe0da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.127+0000 7fe0da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.127+0000 7fe0da7fc640 10 monclient: dump: 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.127+0000 7fe0da7fc640 1 -- 172.21.15.110:0/3819713992 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d400c040 con 0x7fe0c4009ff0 2024-09-15T03:59:06.148 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.127+0000 7fe0da7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:59:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.127+0000 7fe0e0d99640 4 set_mon_vals no callback set 2024-09-15T03:59:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.128+0000 7fe0da7fc640 1 -- 172.21.15.110:0/3819713992 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fe0d4021c50 con 0x7fe0c4009ff0 2024-09-15T03:59:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.128+0000 7fe0da7fc640 1 --2- 172.21.15.110:0/3819713992 >> v2:172.21.15.177:6812/28201 conn(0x7fe0c40414a0 0x7fe0c4043950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.128+0000 7fe0da7fc640 1 -- 172.21.15.110:0/3819713992 <== mon.2 v2:172.21.15.110:3301/0 4 ==== osd_map(21..21 src has 1..21) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fe0d40511f0 con 0x7fe0c4009ff0 2024-09-15T03:59:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.129+0000 7fe0dbfff640 1 --2- 172.21.15.110:0/3819713992 >> v2:172.21.15.177:6812/28201 conn(0x7fe0c40414a0 0x7fe0c4043950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.129+0000 7fe0dbfff640 10 monclient: get_auth_request con 0x7fe0c40414a0 auth_method 0 2024-09-15T03:59:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.131+0000 7fe0dbfff640 1 --2- 172.21.15.110:0/3819713992 >> v2:172.21.15.177:6812/28201 conn(0x7fe0c40414a0 0x7fe0c4043950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc004270 tx=0x7fe0cc034000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:59:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.132+0000 7fe0da7fc640 1 -- 172.21.15.110:0/3819713992 <== mon.2 v2:172.21.15.110:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fe0d4038450 con 0x7fe0c4009ff0 2024-09-15T03:59:06.149 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.132+0000 7fe0da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:59:06.150 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.132+0000 7fe0da7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:59:06.268 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29078]: starting. 2024-09-15T03:59:06.268 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29078]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29077 2024-09-15T03:59:06.268 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29078]: created object 0... 2024-09-15T03:59:06.268 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29078]: created object 25... 2024-09-15T03:59:06.268 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29078]: created object 49... 2024-09-15T03:59:06.268 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29078]: finishing. 2024-09-15T03:59:06.268 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29078]: shutting down. 2024-09-15T03:59:06.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.365+0000 7fe0e8d00640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:59:06.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.365+0000 7fe0e8d00640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:59:06.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.365+0000 7fe0e8d00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:59:06.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.365+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3819713992 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fe0e4070950 con 0x7fe0c4009ff0 2024-09-15T03:59:06.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.365+0000 7fe0da7fc640 1 -- 172.21.15.110:0/3819713992 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7fe0d4036400 con 0x7fe0c4009ff0 2024-09-15T03:59:06.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.365+0000 7fe0da7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:59:06.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.365+0000 7fe0da7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:59:06.367 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:59:06.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.367+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3819713992 >> v2:172.21.15.177:6812/28201 conn(0x7fe0c40414a0 msgr2=0x7fe0c4043950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.367+0000 7fe0e8d00640 1 --2- 172.21.15.110:0/3819713992 >> v2:172.21.15.177:6812/28201 conn(0x7fe0c40414a0 0x7fe0c4043950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc004270 tx=0x7fe0cc034000 comp rx=0 tx=0).stop 2024-09-15T03:59:06.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.367+0000 7fe0e8d00640 10 monclient: shutdown 2024-09-15T03:59:06.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.367+0000 7fe0e8d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:59:06.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.367+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4009ff0 msgr2=0x7fe0c400a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.367+0000 7fe0e8d00640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4009ff0 0x7fe0c400a3c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4016c70 tx=0x7fe0d4027000 comp rx=0 tx=0).stop 2024-09-15T03:59:06.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.367+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3819713992 shutdown_connections 2024-09-15T03:59:06.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.367+0000 7fe0e8d00640 1 --2- 172.21.15.110:0/3819713992 >> v2:172.21.15.177:6812/28201 conn(0x7fe0c40414a0 0x7fe0c4043950 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.367+0000 7fe0e8d00640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe0d0004cc0 0x7fe0c4009750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.367+0000 7fe0e8d00640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe0d00039d0 0x7fe0c4008d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.367+0000 7fe0e8d00640 1 --2- 172.21.15.110:0/3819713992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe0c4009ff0 0x7fe0c400a3c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.367+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3819713992 >> 172.21.15.110:0/3819713992 conn(0x7fe0e4062dd0 msgr2=0x7fe0e40a4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:59:06.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.367+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3819713992 shutdown_connections 2024-09-15T03:59:06.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.367+0000 7fe0e8d00640 1 -- 172.21.15.110:0/3819713992 wait complete. 2024-09-15T03:59:06.379 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:59:06.379 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:59:06.379 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:59:06.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.471+0000 7f751019d640 10 monclient: get_monmap_and_config 2024-09-15T03:59:06.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.471+0000 7f751019d640 10 monclient: build_initial_monmap 2024-09-15T03:59:06.472 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.471+0000 7f751019d640 10 monclient: monmap: 2024-09-15T03:59:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:59:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:59:06.472728+0000 2024-09-15T03:59:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:59:06.472728+0000 2024-09-15T03:59:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:59:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:59:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:59:06.473 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:59:06.474 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:59:06.474 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:59:06.474 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 1 Processor -- start 2024-09-15T03:59:06.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 1 -- start start 2024-09-15T03:59:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 10 monclient: init 2024-09-15T03:59:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 10 monclient: _reopen_session rank -1 2024-09-15T03:59:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-15T03:59:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f75080a1cf0 0x7f750809e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 10 monclient(hunting): picked mon.noname-a con 0x7f75080a1cf0 addr v2:172.21.15.110:3300/0 2024-09-15T03:59:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7508131a80 0x7f750809eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 10 monclient(hunting): picked mon.noname-c con 0x7f7508131a80 addr v2:172.21.15.177:3300/0 2024-09-15T03:59:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f750809f760 addr v1:172.21.15.110:6789/0 2024-09-15T03:59:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750809cb70 con 0x7f750809f760 2024-09-15T03:59:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75080a20c0 con 0x7f75080a1cf0 2024-09-15T03:59:06.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750809c330 con 0x7f7508131a80 2024-09-15T03:59:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 10 monclient(hunting): _renew_subs 2024-09-15T03:59:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f751019d640 10 monclient(hunting): authenticate will time out at 770.116973s 2024-09-15T03:59:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f750df39640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f75080a1cf0 msgr2=0x7f750809e660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:59:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.472+0000 7f750e73a640 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f750809f760 legacy=0x7f750809fb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2024-09-15T03:59:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750df39640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f75080a1cf0 0x7f750809e660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:59:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7508131a80 0x7f750809eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7508131a80 0x7f750809eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.110:53122/0 (socket says 172.21.15.110:53122) 2024-09-15T03:59:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 1 -- 172.21.15.110:0/2229503234 learned_addr learned my addr 172.21.15.110:0/2229503234 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:59:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 10 monclient(hunting): get_auth_request con 0x7f7508131a80 auth_method 0 2024-09-15T03:59:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.478 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:06.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:59:06.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:59:06.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:59:06.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 10 monclient(hunting): handle_auth_done global_id 4354 payload 274 2024-09-15T03:59:06.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 1 -- 172.21.15.110:0/2229503234 >> v2:172.21.15.110:3300/0 conn(0x7f75080a1cf0 msgr2=0x7f750809e660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:59:06.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 1 --2- 172.21.15.110:0/2229503234 >> v2:172.21.15.110:3300/0 conn(0x7f75080a1cf0 0x7f750809e660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 1 -- 172.21.15.110:0/2229503234 >> v1:172.21.15.110:6789/0 conn(0x7f750809f760 legacy=0x7f750809fb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:59:06.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 10 monclient: _finish_hunting 0 2024-09-15T03:59:06.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 1 monclient: found mon.noname-c 2024-09-15T03:59:06.479 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:59:06.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:59:06.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.473+0000 7f750d738640 1 -- 172.21.15.110:0/2229503234 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7508131fb0 con 0x7f7508131a80 2024-09-15T03:59:06.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f750d738640 1 -- 172.21.15.110:0/2229503234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7508131a80 msgr2=0x7f750809eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:59:06.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f750d738640 1 --2- 172.21.15.110:0/2229503234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7508131a80 0x7f750809eba0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7508068dc0 tx=0x7f74f80091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=125819dbe2b36210 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:59:06.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f750cf37640 1 -- 172.21.15.110:0/2229503234 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8032440 con 0x7f7508131a80 2024-09-15T03:59:06.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f750cf37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f750cf37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:59:06.487 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f750cf37640 10 monclient: dump: 2024-09-15T03:59:06.487 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.488 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f750cf37640 10 monclient: _finish_auth 0 2024-09-15T03:59:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f750cf37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:59:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f750cf37640 1 -- 172.21.15.110:0/2229503234 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f80325a0 con 0x7f7508131a80 2024-09-15T03:59:06.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f751019d640 5 monclient: authenticate success, global_id 4354 2024-09-15T03:59:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f751019d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:59:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f750cf37640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:59:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f751019d640 10 monclient: get_monmap_and_config success 2024-09-15T03:59:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f750cf37640 1 -- 172.21.15.110:0/2229503234 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8032940 con 0x7f7508131a80 2024-09-15T03:59:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f750cf37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f751019d640 4 set_mon_vals no callback set 2024-09-15T03:59:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f750cf37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:59:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.476+0000 7f750cf37640 10 monclient: dump: 2024-09-15T03:59:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.489 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.477+0000 7f751019d640 10 monclient: shutdown 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.477+0000 7f751019d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.477+0000 7f751019d640 1 -- 172.21.15.110:0/2229503234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7508131a80 msgr2=0x7f750809eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.477+0000 7f751019d640 1 --2- 172.21.15.110:0/2229503234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7508131a80 0x7f750809eba0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7508068dc0 tx=0x7f74f80091d0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.477+0000 7f751019d640 1 -- 172.21.15.110:0/2229503234 shutdown_connections 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.477+0000 7f751019d640 1 --2- 172.21.15.110:0/2229503234 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7508131a80 0x7f750809eba0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.477+0000 7f751019d640 1 --2- 172.21.15.110:0/2229503234 >> v2:172.21.15.110:3300/0 conn(0x7f75080a1cf0 0x7f750809e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.477+0000 7f751019d640 1 -- 172.21.15.110:0/2229503234 >> 172.21.15.110:0/2229503234 conn(0x7f7508062f70 msgr2=0x7f7508063340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.477+0000 7f751019d640 1 -- 172.21.15.110:0/2229503234 shutdown_connections 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.477+0000 7f751019d640 1 -- 172.21.15.110:0/2229503234 wait complete. 2024-09-15T03:59:06.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 10 monclient: build_initial_monmap 2024-09-15T03:59:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 10 monclient: monmap: 2024-09-15T03:59:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:59:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:59:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:59:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:59:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:59:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:59:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:59:06.491 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:59:06.491 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 1 Processor -- start 2024-09-15T03:59:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 1 -- start start 2024-09-15T03:59:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 10 monclient: init 2024-09-15T03:59:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 10 monclient: _reopen_session rank -1 2024-09-15T03:59:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:59:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f750809f760 0x7f750809a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 10 monclient(hunting): picked mon.noname-b con 0x7f750809f760 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:59:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75080a1cf0 0x7f7508130750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 10 monclient(hunting): picked mon.noname-c con 0x7f75080a1cf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:59:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7508131a80 0x7f7508140c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 10 monclient(hunting): picked mon.noname-a con 0x7f7508131a80 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:59:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750811db50 con 0x7f7508131a80 2024-09-15T03:59:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750809b490 con 0x7f750809f760 2024-09-15T03:59:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750811d870 con 0x7f75080a1cf0 2024-09-15T03:59:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 10 monclient(hunting): _renew_subs 2024-09-15T03:59:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f751019d640 10 monclient(hunting): authenticate will time out at 770.122856s 2024-09-15T03:59:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f750e73a640 1 -- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7508131a80 msgr2=0x7f7508140c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:59:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f750e73a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7508131a80 0x7f7508140c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:59:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f750d738640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75080a1cf0 0x7f7508130750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.478+0000 7f750d738640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75080a1cf0 0x7f7508130750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:51742/0 (socket says 172.21.15.110:51742) 2024-09-15T03:59:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 1 -- 172.21.15.110:0/4188082566 learned_addr learned my addr 172.21.15.110:0/4188082566 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:59:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 10 monclient(hunting): get_auth_request con 0x7f75080a1cf0 auth_method 0 2024-09-15T03:59:06.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750df39640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f750809f760 0x7f750809a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750df39640 10 monclient(hunting): get_auth_request con 0x7f750809f760 auth_method 0 2024-09-15T03:59:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750df39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750df39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750df39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:59:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:59:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:59:06.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750df39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:59:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750df39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:59:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750df39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:59:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 10 monclient(hunting): handle_auth_done global_id 4412 payload 824 2024-09-15T03:59:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 1 -- 172.21.15.110:0/4188082566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f750809f760 msgr2=0x7f750809a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f750809f760 0x7f750809a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 1 -- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7508131a80 msgr2=0x7f7508140c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:59:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7508131a80 0x7f7508140c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 10 monclient: _finish_hunting 0 2024-09-15T03:59:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 1 monclient: found mon.noname-c 2024-09-15T03:59:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:59:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:59:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.479+0000 7f750d738640 1 -- 172.21.15.110:0/4188082566 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f8009450 con 0x7f75080a1cf0 2024-09-15T03:59:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f750d738640 1 -- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75080a1cf0 msgr2=0x7f7508130750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:59:06.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f750d738640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75080a1cf0 0x7f7508130750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f74f8009210 tx=0x7f74f8032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:59:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f74fdffb640 1 -- 172.21.15.110:0/4188082566 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f803c960 con 0x7f75080a1cf0 2024-09-15T03:59:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f74fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f74fdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:59:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f74fdffb640 10 monclient: dump: 2024-09-15T03:59:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.496 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f74fdffb640 10 monclient: _finish_auth 0 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f74fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f74fdffb640 1 -- 172.21.15.110:0/4188082566 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f803cac0 con 0x7f75080a1cf0 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f751019d640 5 monclient: authenticate success, global_id 4412 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f74fdffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f751019d640 10 monclient: _renew_subs 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f74fdffb640 1 -- 172.21.15.110:0/4188082566 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f803ce40 con 0x7f75080a1cf0 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f751019d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f751019d640 1 -- 172.21.15.110:0/4188082566 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750811de30 con 0x7f75080a1cf0 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f74fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f74fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f74fdffb640 10 monclient: dump: 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.497 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f74fffff640 4 set_mon_vals no callback set 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f751019d640 10 monclient: _renew_subs 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f751019d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.481+0000 7f751019d640 1 -- 172.21.15.110:0/4188082566 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750811c980 con 0x7f75080a1cf0 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f751019d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f751019d640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508130c90 0x7f75081411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f751019d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f751019d640 1 -- 172.21.15.110:0/4188082566 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750806f1e0 con 0x7f7508130c90 2024-09-15T03:59:06.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f74fdffb640 1 -- 172.21.15.110:0/4188082566 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f74f8039040 con 0x7f75080a1cf0 2024-09-15T03:59:06.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f750df39640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508130c90 0x7f75081411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f750df39640 10 monclient: get_auth_request con 0x7f7508130c90 auth_method 0 2024-09-15T03:59:06.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f750df39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f750df39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f750df39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:06.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f750df39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:59:06.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f750df39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:59:06.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f74fdffb640 1 --2- 172.21.15.110:0/4188082566 >> v2:172.21.15.177:6812/28201 conn(0x7f74ec0386e0 0x7f74ec03ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f750df39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:59:06.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.482+0000 7f74fdffb640 1 -- 172.21.15.110:0/4188082566 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f74f806b7b0 con 0x7f75080a1cf0 2024-09-15T03:59:06.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.483+0000 7f750df39640 10 monclient(hunting): handle_auth_done global_id 4415 payload 824 2024-09-15T03:59:06.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.483+0000 7f750df39640 1 -- 172.21.15.110:0/4188082566 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f750809b9d0 con 0x7f7508130c90 2024-09-15T03:59:06.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.483+0000 7f750e73a640 1 --2- 172.21.15.110:0/4188082566 >> v2:172.21.15.177:6812/28201 conn(0x7f74ec0386e0 0x7f74ec03ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.483+0000 7f750e73a640 10 monclient: get_auth_request con 0x7f74ec0386e0 auth_method 0 2024-09-15T03:59:06.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.485+0000 7f750df39640 1 -- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508130c90 msgr2=0x7f75081411c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:59:06.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.485+0000 7f750df39640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508130c90 0x7f75081411c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f75040095e0 tx=0x7f75040090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:59:06.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.485+0000 7f74fdffb640 1 -- 172.21.15.110:0/4188082566 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750400e9f0 con 0x7f7508130c90 2024-09-15T03:59:06.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.485+0000 7f74fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.485+0000 7f74fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:59:06.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.485+0000 7f74fdffb640 10 monclient: dump: 2024-09-15T03:59:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.501 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.485+0000 7f750e73a640 1 --2- 172.21.15.110:0/4188082566 >> v2:172.21.15.177:6812/28201 conn(0x7f74ec0386e0 0x7f74ec03ab90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f750811e600 tx=0x7f7500009370 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:59:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.485+0000 7f74fdffb640 1 -- 172.21.15.110:0/4188082566 <== mon.2 v2:172.21.15.110:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f750400eb50 con 0x7f7508130c90 2024-09-15T03:59:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.485+0000 7f74fdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:59:06.501 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.485+0000 7f74fdffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:59:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.485+0000 7f74fdffb640 1 -- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508130c90 msgr2=0x7f75081411c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.485+0000 7f74fdffb640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508130c90 0x7f75081411c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f75040095e0 tx=0x7f75040090f0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.485+0000 7f74fdffb640 1 -- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508130c90 msgr2=0x7f75081411c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:59:06.502 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.485+0000 7f74fdffb640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508130c90 0x7f75081411c0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f75040095e0 tx=0x7f75040090f0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.491+0000 7f751019d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:59:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.491+0000 7f751019d640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508141700 0x7f7508132720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.491+0000 7f751019d640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.503 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.491+0000 7f751019d640 1 -- 172.21.15.110:0/4188082566 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750811dfc0 con 0x7f7508141700 2024-09-15T03:59:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.491+0000 7f750df39640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508141700 0x7f7508132720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.491+0000 7f750df39640 10 monclient: get_auth_request con 0x7f7508141700 auth_method 0 2024-09-15T03:59:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.491+0000 7f750df39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.491+0000 7f750df39640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.491+0000 7f750df39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.491+0000 7f750df39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:59:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.491+0000 7f750df39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:59:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.491+0000 7f750df39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:59:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.491+0000 7f750df39640 10 monclient(hunting): handle_auth_done global_id 4418 payload 824 2024-09-15T03:59:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.491+0000 7f750df39640 1 -- 172.21.15.110:0/4188082566 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f750811dcc0 con 0x7f7508141700 2024-09-15T03:59:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.493+0000 7f750df39640 1 -- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508141700 msgr2=0x7f7508132720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:59:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.493+0000 7f750df39640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508141700 0x7f7508132720 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7504017000 tx=0x7f7504015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:59:06.504 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.493+0000 7f74fdffb640 1 -- 172.21.15.110:0/4188082566 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7504025070 con 0x7f7508141700 2024-09-15T03:59:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.493+0000 7f74fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.493+0000 7f74fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:59:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.493+0000 7f74fdffb640 10 monclient: dump: 2024-09-15T03:59:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.505 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.505 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f74fdffb640 1 -- 172.21.15.110:0/4188082566 <== mon.2 v2:172.21.15.110:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4566 (secure 0 0 0) 0x7f750401b660 con 0x7f7508141700 2024-09-15T03:59:06.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f74fdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:59:06.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f74fdffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:59:06.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f74fdffb640 1 -- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508141700 msgr2=0x7f7508132720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f74fdffb640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508141700 0x7f7508132720 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7504017000 tx=0x7f7504015000 comp rx=0 tx=0).stop 2024-09-15T03:59:06.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f74fdffb640 1 -- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508141700 msgr2=0x7f7508132720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:59:06.506 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f74fdffb640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7508141700 0x7f7508132720 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7504017000 tx=0x7f7504015000 comp rx=0 tx=0).stop 2024-09-15T03:59:06.506 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T03:59:06.506 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T03:59:06.506 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T03:59:06.507 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T03:59:06.507 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 12, 2024-09-15T03:59:06.507 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T03:59:06.507 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T03:59:06.507 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T03:59:06.507 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:59:06.507 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 0, 2024-09-15T03:59:06.507 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T03:59:06.507 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T03:59:06.507 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T03:59:06.507 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:59:06.507 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:59:06.507 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:59:06.507 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:59:06.508 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:59:06.508 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:59:06.508 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:59:06.508 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:59:06.508 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:59:06.508 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:59:06.508 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:59:06.508 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T03:59:06.508 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:59:06.508 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:59:06.508 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:59:06.508 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:59:06.508 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:59:06.508 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:59:06.509 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:59:06.509 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:59:06.509 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:59:06.509 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:59:06.509 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.509 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T03:59:06.509 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T03:59:06.509 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T03:59:06.509 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T03:59:06.509 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T03:59:06.509 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T03:59:06.509 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T03:59:06.510 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T03:59:06.510 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T03:59:06.510 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:59:06.510 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T03:59:06.510 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T03:59:06.510 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T03:59:06.510 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T03:59:06.510 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T03:59:06.510 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T03:59:06.510 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T03:59:06.510 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:59:06.510 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:59:06.510 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:59:06.511 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:59:06.511 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:59:06.511 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:59:06.511 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:59:06.511 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:59:06.511 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:59:06.511 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:59:06.511 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T03:59:06.511 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.511 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T03:59:06.511 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.511 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T03:59:06.511 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T03:59:06.511 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:59:06.512 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:59:06.512 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.512 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:59:06.512 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T03:59:06.512 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:59:06.512 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.512 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.512 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:59:06.512 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T03:59:06.512 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:59:06.512 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.512 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:59:06.512 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.512 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T03:59:06.513 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T03:59:06.513 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:59:06.513 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:59:06.513 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:59:06.513 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.513 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.513 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T03:59:06.513 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T03:59:06.513 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:59:06.513 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:59:06.513 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.513 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:59:06.513 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T03:59:06.513 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:59:06.514 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.514 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.514 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:59:06.514 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T03:59:06.514 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:59:06.514 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.514 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:59:06.514 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.514 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T03:59:06.514 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T03:59:06.514 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:59:06.514 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:59:06.514 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:59:06.514 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:59:06.515 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.516 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:59:06.516 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.516 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T03:59:06.516 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T03:59:06.516 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:59:06.516 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:59:06.516 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:59:06.516 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.516 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:59:06.516 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.516 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T03:59:06.516 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T03:59:06.516 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.517 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:59:06.517 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:59:06.517 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T03:59:06.517 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.517 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:59:06.517 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T03:59:06.517 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.517 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:59:06.517 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:59:06.517 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 3 2024-09-15T03:59:06.517 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.517 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:59:06.517 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T03:59:06.518 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.518 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:59:06.518 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:59:06.518 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 2 2024-09-15T03:59:06.518 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.518 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:59:06.518 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.518 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T03:59:06.518 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T03:59:06.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f751019d640 1 -- 172.21.15.110:0/4188082566 >> v2:172.21.15.177:6812/28201 conn(0x7f74ec0386e0 msgr2=0x7f74ec03ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.519 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f751019d640 1 --2- 172.21.15.110:0/4188082566 >> v2:172.21.15.177:6812/28201 conn(0x7f74ec0386e0 0x7f74ec03ab90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f750811e600 tx=0x7f7500009370 comp rx=0 tx=0).stop 2024-09-15T03:59:06.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f751019d640 10 monclient: shutdown 2024-09-15T03:59:06.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f751019d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:59:06.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f751019d640 1 -- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75080a1cf0 msgr2=0x7f7508130750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f751019d640 1 --2- 172.21.15.110:0/4188082566 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f75080a1cf0 0x7f7508130750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f74f8009210 tx=0x7f74f8032000 comp rx=0 tx=0).stop 2024-09-15T03:59:06.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f750e73a640 1 -- 172.21.15.110:0/4188082566 reap_dead start 2024-09-15T03:59:06.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f750e73a640 1 -- 172.21.15.110:0/4188082566 reap_dead start 2024-09-15T03:59:06.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f751019d640 1 -- 172.21.15.110:0/4188082566 shutdown_connections 2024-09-15T03:59:06.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f751019d640 1 -- 172.21.15.110:0/4188082566 >> 172.21.15.110:0/4188082566 conn(0x7f7508062f70 msgr2=0x7f7508134940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:59:06.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f751019d640 1 -- 172.21.15.110:0/4188082566 shutdown_connections 2024-09-15T03:59:06.520 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.494+0000 7f751019d640 1 -- 172.21.15.110:0/4188082566 wait complete. 2024-09-15T03:59:06.520 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:59:06.521 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:59:06.521 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:59:06.614 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.613+0000 7f1376a80640 10 monclient: get_monmap_and_config 2024-09-15T03:59:06.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.613+0000 7f1376a80640 10 monclient: build_initial_monmap 2024-09-15T03:59:06.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.613+0000 7f1376a80640 10 monclient: monmap: 2024-09-15T03:59:06.615 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:59:06.615 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.615 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:59:06.614913+0000 2024-09-15T03:59:06.615 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:59:06.614913+0000 2024-09-15T03:59:06.615 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:59:06.615 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.615 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T03:59:06.616 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T03:59:06.616 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T03:59:06.616 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T03:59:06.616 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T03:59:06.616 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.614+0000 7f1376a80640 1 Processor -- start 2024-09-15T03:59:06.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.614+0000 7f1376a80640 1 -- start start 2024-09-15T03:59:06.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.614+0000 7f1376a80640 10 monclient: init 2024-09-15T03:59:06.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1376a80640 10 monclient: _reopen_session rank -1 2024-09-15T03:59:06.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1376a80640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-15T03:59:06.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1376a80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13700a3480 addr v1:172.21.15.177:6789/0 2024-09-15T03:59:06.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1376a80640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f137006e8d0 0x7f137012fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1376a80640 10 monclient(hunting): picked mon.noname-c con 0x7f137006e8d0 addr v2:172.21.15.177:3300/0 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1376a80640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1370134870 0x7f1370136c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1376a80640 10 monclient(hunting): picked mon.noname-a con 0x7f1370134870 addr v2:172.21.15.110:3300/0 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1376a80640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13700a3850 con 0x7f13700a3480 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1376a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1376a80640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f137009f050 con 0x7f1370134870 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1376a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1376a80640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f137006ed10 con 0x7f137006e8d0 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1376a80640 10 monclient(hunting): _renew_subs 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1376a80640 10 monclient(hunting): authenticate will time out at 770.259248s 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f137501d640 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f1370134870 msgr2=0x7f1370136c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f137501d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1370134870 0x7f1370136c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f137481c640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f13700a3480 0x7f137012f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.110:50676/0 (socket says 172.21.15.110:50676) 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f137481c640 1 -- 172.21.15.110:0/3762823592 learned_addr learned my addr 172.21.15.110:0/3762823592 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f137006e8d0 0x7f137012fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 10 monclient(hunting): get_auth_request con 0x7f137006e8d0 auth_method 0 2024-09-15T03:59:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:59:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:59:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:59:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 10 monclient(hunting): handle_auth_done global_id 4360 payload 274 2024-09-15T03:59:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 1 -- 172.21.15.110:0/3762823592 >> v2:172.21.15.110:3300/0 conn(0x7f1370134870 msgr2=0x7f1370136c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:59:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 1 --2- 172.21.15.110:0/3762823592 >> v2:172.21.15.110:3300/0 conn(0x7f1370134870 0x7f1370136c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 1 -- 172.21.15.110:0/3762823592 >> v1:172.21.15.177:6789/0 conn(0x7f13700a3480 legacy=0x7f137012f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:59:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 10 monclient: _finish_hunting 0 2024-09-15T03:59:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 1 monclient: found mon.noname-c 2024-09-15T03:59:06.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:59:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T03:59:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.615+0000 7f1367fff640 1 -- 172.21.15.110:0/3762823592 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f137006d590 con 0x7f137006e8d0 2024-09-15T03:59:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.618+0000 7f1367fff640 1 -- 172.21.15.110:0/3762823592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f137006e8d0 msgr2=0x7f137012fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:59:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.618+0000 7f1367fff640 1 --2- 172.21.15.110:0/3762823592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f137006e8d0 0x7f137012fcf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f137009cfc0 tx=0x7f13600090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ce182601eeb7ab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:59:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.618+0000 7f13677fe640 1 -- 172.21.15.110:0/3762823592 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360032440 con 0x7f137006e8d0 2024-09-15T03:59:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.618+0000 7f13677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f13677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:59:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f13677fe640 10 monclient: dump: 2024-09-15T03:59:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.624 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.625 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.625 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.625 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.625 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.625 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.625 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f13677fe640 10 monclient: _finish_auth 0 2024-09-15T03:59:06.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f13677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:59:06.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f1376a80640 5 monclient: authenticate success, global_id 4360 2024-09-15T03:59:06.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f1376a80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:59:06.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f13677fe640 1 -- 172.21.15.110:0/3762823592 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13600325a0 con 0x7f137006e8d0 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f13677fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f1376a80640 10 monclient: get_monmap_and_config success 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f1376a80640 4 set_mon_vals no callback set 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f13677fe640 1 -- 172.21.15.110:0/3762823592 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360032940 con 0x7f137006e8d0 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f13677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f13677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f13677fe640 10 monclient: dump: 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.626 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.627 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f1376a80640 10 monclient: shutdown 2024-09-15T03:59:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f1376a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:59:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f1376a80640 1 -- 172.21.15.110:0/3762823592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f137006e8d0 msgr2=0x7f137012fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f1376a80640 1 --2- 172.21.15.110:0/3762823592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f137006e8d0 0x7f137012fcf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f137009cfc0 tx=0x7f13600090f0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f1376a80640 1 -- 172.21.15.110:0/3762823592 shutdown_connections 2024-09-15T03:59:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f1376a80640 1 --2- 172.21.15.110:0/3762823592 >> v2:172.21.15.110:3300/0 conn(0x7f1370134870 0x7f1370136c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f1376a80640 1 --2- 172.21.15.110:0/3762823592 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f137006e8d0 0x7f137012fcf0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f1376a80640 1 -- 172.21.15.110:0/3762823592 >> 172.21.15.110:0/3762823592 conn(0x7f13700635b0 msgr2=0x7f13700a2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:59:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f1376a80640 1 -- 172.21.15.110:0/3762823592 shutdown_connections 2024-09-15T03:59:06.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.619+0000 7f1376a80640 1 -- 172.21.15.110:0/3762823592 wait complete. 2024-09-15T03:59:06.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 10 monclient: build_initial_monmap 2024-09-15T03:59:06.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 10 monclient: monmap: 2024-09-15T03:59:06.628 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T03:59:06.628 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:59:06.628 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T03:59:06.628 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T03:59:06.628 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T03:59:06.628 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.628 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T03:59:06.629 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T03:59:06.629 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T03:59:06.629 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 1 Processor -- start 2024-09-15T03:59:06.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 1 -- start start 2024-09-15T03:59:06.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 10 monclient: init 2024-09-15T03:59:06.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 10 monclient: _reopen_session rank -1 2024-09-15T03:59:06.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:59:06.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f137006e8d0 0x7f137012c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 10 monclient(hunting): picked mon.noname-c con 0x7f137006e8d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T03:59:06.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13700a3480 0x7f1370098a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 10 monclient(hunting): picked mon.noname-b con 0x7f13700a3480 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T03:59:06.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1370134870 0x7f1370098f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 10 monclient(hunting): picked mon.noname-a con 0x7f1370134870 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T03:59:06.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137013fbb0 con 0x7f1370134870 2024-09-15T03:59:06.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137009ab30 con 0x7f13700a3480 2024-09-15T03:59:06.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137006c8e0 con 0x7f137006e8d0 2024-09-15T03:59:06.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 10 monclient(hunting): _renew_subs 2024-09-15T03:59:06.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f1376a80640 10 monclient(hunting): authenticate will time out at 770.264945s 2024-09-15T03:59:06.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.620+0000 7f137501d640 1 -- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1370134870 msgr2=0x7f1370098f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3300/0 2024-09-15T03:59:06.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137501d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1370134870 0x7f1370098f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T03:59:06.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f137006e8d0 0x7f137012c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f137006e8d0 0x7f137012c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:51764/0 (socket says 172.21.15.110:51764) 2024-09-15T03:59:06.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 1 -- 172.21.15.110:0/2965403607 learned_addr learned my addr 172.21.15.110:0/2965403607 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T03:59:06.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 10 monclient(hunting): get_auth_request con 0x7f137006e8d0 auth_method 0 2024-09-15T03:59:06.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:06.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:59:06.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:59:06.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:59:06.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f1367fff640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13700a3480 0x7f1370098a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f1367fff640 10 monclient(hunting): get_auth_request con 0x7f13700a3480 auth_method 0 2024-09-15T03:59:06.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f1367fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f1367fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f1367fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:06.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 10 monclient(hunting): handle_auth_done global_id 4421 payload 824 2024-09-15T03:59:06.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 1 -- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13700a3480 msgr2=0x7f1370098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13700a3480 0x7f1370098a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 1 -- 172.21.15.110:0/2965403607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1370134870 msgr2=0x7f1370098f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:59:06.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1370134870 0x7f1370098f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 10 monclient: _finish_hunting 0 2024-09-15T03:59:06.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 1 monclient: found mon.noname-c 2024-09-15T03:59:06.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:59:06.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T03:59:06.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f137481c640 1 -- 172.21.15.110:0/2965403607 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1360009390 con 0x7f137006e8d0 2024-09-15T03:59:06.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.621+0000 7f1367fff640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f13700a3480 0x7f1370098a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T03:59:06.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f137481c640 1 -- 172.21.15.110:0/2965403607 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f137006e8d0 msgr2=0x7f137012c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:59:06.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f137481c640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f137006e8d0 0x7f137012c570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f136c00ccd0 tx=0x7f136c0053f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:59:06.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f137481c640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f137006e8d0 0x7f137012c570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f136c00ccd0 tx=0x7f136c0053f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f136c00aa60 mon_map magic: 0 v1 2024-09-15T03:59:06.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1364ff9640 1 -- 172.21.15.110:0/2965403607 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c00aa60 con 0x7f137006e8d0 2024-09-15T03:59:06.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1364ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1364ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:59:06.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1364ff9640 10 monclient: dump: 2024-09-15T03:59:06.639 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.639 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.639 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.640 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.640 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.640 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.640 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.640 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.640 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.640 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1364ff9640 10 monclient: _finish_auth 0 2024-09-15T03:59:06.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1364ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:59:06.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1364ff9640 1 -- 172.21.15.110:0/2965403607 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136c00abe0 con 0x7f137006e8d0 2024-09-15T03:59:06.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1376a80640 5 monclient: authenticate success, global_id 4421 2024-09-15T03:59:06.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1376a80640 10 monclient: _renew_subs 2024-09-15T03:59:06.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1376a80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:59:06.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1376a80640 1 -- 172.21.15.110:0/2965403607 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137012eca0 con 0x7f137006e8d0 2024-09-15T03:59:06.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1364ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:59:06.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1376a80640 10 monclient: _renew_subs 2024-09-15T03:59:06.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1376a80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T03:59:06.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1376a80640 1 -- 172.21.15.110:0/2965403607 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137009b180 con 0x7f137006e8d0 2024-09-15T03:59:06.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.623+0000 7f1366ffd640 4 set_mon_vals no callback set 2024-09-15T03:59:06.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.624+0000 7f1376a80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:59:06.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.624+0000 7f1376a80640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f137012cab0 0x7f13700994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.624+0000 7f1376a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.624+0000 7f1376a80640 1 -- 172.21.15.110:0/2965403607 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1370067ab0 con 0x7f137012cab0 2024-09-15T03:59:06.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.624+0000 7f1364ff9640 1 -- 172.21.15.110:0/2965403607 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c0070f0 con 0x7f137006e8d0 2024-09-15T03:59:06.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.624+0000 7f1364ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.624+0000 7f1364ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:59:06.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.624+0000 7f1364ff9640 10 monclient: dump: 2024-09-15T03:59:06.644 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.645 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.645 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.645 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.645 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.645 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.645 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.645 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.645 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.645 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.624+0000 7f1364ff9640 1 -- 172.21.15.110:0/2965403607 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f136c0074e0 con 0x7f137006e8d0 2024-09-15T03:59:06.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.624+0000 7f1364ff9640 1 --2- 172.21.15.110:0/2965403607 >> v2:172.21.15.177:6812/28201 conn(0x7f135c038700 0x7f135c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.625+0000 7f137501d640 1 --2- 172.21.15.110:0/2965403607 >> v2:172.21.15.177:6812/28201 conn(0x7f135c038700 0x7f135c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.625+0000 7f137501d640 10 monclient: get_auth_request con 0x7f135c038700 auth_method 0 2024-09-15T03:59:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.625+0000 7f1367fff640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f137012cab0 0x7f13700994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.625+0000 7f1367fff640 10 monclient: get_auth_request con 0x7f137012cab0 auth_method 0 2024-09-15T03:59:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.625+0000 7f1367fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.625+0000 7f1367fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.625+0000 7f1367fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.625+0000 7f1367fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:59:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.625+0000 7f1367fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:59:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.625+0000 7f1367fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:59:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.625+0000 7f1367fff640 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2024-09-15T03:59:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.625+0000 7f1367fff640 1 -- 172.21.15.110:0/2965403607 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f137009a6d0 con 0x7f137012cab0 2024-09-15T03:59:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.627+0000 7f137501d640 1 --2- 172.21.15.110:0/2965403607 >> v2:172.21.15.177:6812/28201 conn(0x7f135c038700 0x7f135c03abb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1370119410 tx=0x7f1368006cb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:59:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.627+0000 7f1367fff640 1 -- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f137012cab0 msgr2=0x7f13700994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:59:06.646 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.627+0000 7f1367fff640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f137012cab0 0x7f13700994c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1360034000 tx=0x7f1360032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:59:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.628+0000 7f1364ff9640 1 -- 172.21.15.110:0/2965403607 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136003ca10 con 0x7f137012cab0 2024-09-15T03:59:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.628+0000 7f1364ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.628+0000 7f1364ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:59:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.628+0000 7f1364ff9640 10 monclient: dump: 2024-09-15T03:59:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.647 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.647 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.628+0000 7f1364ff9640 1 -- 172.21.15.110:0/2965403607 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f136003cb70 con 0x7f137012cab0 2024-09-15T03:59:06.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.628+0000 7f1364ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:59:06.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.628+0000 7f1364ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T03:59:06.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.628+0000 7f1364ff9640 1 -- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f137012cab0 msgr2=0x7f13700994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.628+0000 7f1364ff9640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f137012cab0 0x7f13700994c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1360034000 tx=0x7f1360032000 comp rx=0 tx=0).stop 2024-09-15T03:59:06.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.628+0000 7f1364ff9640 1 -- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f137012cab0 msgr2=0x7f13700994c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:59:06.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.628+0000 7f1364ff9640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f137012cab0 0x7f13700994c0 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1360034000 tx=0x7f1360032000 comp rx=0 tx=0).stop 2024-09-15T03:59:06.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.633+0000 7f1376a80640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:59:06.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.634+0000 7f1376a80640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1370099a00 0x7f1370095a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T03:59:06.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.634+0000 7f1376a80640 10 monclient(hunting): start opening mon connection 2024-09-15T03:59:06.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.634+0000 7f1376a80640 1 -- 172.21.15.110:0/2965403607 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137012ce80 con 0x7f1370099a00 2024-09-15T03:59:06.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.634+0000 7f1367fff640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1370099a00 0x7f1370095a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T03:59:06.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.634+0000 7f1367fff640 10 monclient: get_auth_request con 0x7f1370099a00 auth_method 0 2024-09-15T03:59:06.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.634+0000 7f1367fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T03:59:06.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.634+0000 7f1367fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T03:59:06.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.634+0000 7f1367fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:59:06.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.634+0000 7f1367fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T03:59:06.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.634+0000 7f1367fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T03:59:06.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.634+0000 7f1367fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T03:59:06.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.634+0000 7f1367fff640 10 monclient(hunting): handle_auth_done global_id 4369 payload 824 2024-09-15T03:59:06.649 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.634+0000 7f1367fff640 1 -- 172.21.15.110:0/2965403607 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f13700647e0 con 0x7f1370099a00 2024-09-15T03:59:06.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1367fff640 1 -- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1370099a00 msgr2=0x7f1370095a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:59:06.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1367fff640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1370099a00 0x7f1370095a70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1370067810 tx=0x7f136003d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T03:59:06.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1364ff9640 1 -- 172.21.15.110:0/2965403607 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360047e30 con 0x7f1370099a00 2024-09-15T03:59:06.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1364ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:59:06.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1364ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:59:06.650 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1364ff9640 10 monclient: dump: 2024-09-15T03:59:06.650 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T03:59:06.650 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T03:59:06.650 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.650 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T03:59:06.651 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T03:59:06.651 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T03:59:06.651 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T03:59:06.651 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T03:59:06.651 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T03:59:06.651 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T03:59:06.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1364ff9640 1 -- 172.21.15.110:0/2965403607 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4738 (secure 0 0 0) 0x7f136003b5e0 con 0x7f1370099a00 2024-09-15T03:59:06.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1364ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:59:06.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1364ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T03:59:06.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1364ff9640 1 -- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1370099a00 msgr2=0x7f1370095a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1364ff9640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1370099a00 0x7f1370095a70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1370067810 tx=0x7f136003d000 comp rx=0 tx=0).stop 2024-09-15T03:59:06.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1364ff9640 1 -- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1370099a00 msgr2=0x7f1370095a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:59:06.651 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1364ff9640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1370099a00 0x7f1370095a70 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1370067810 tx=0x7f136003d000 comp rx=0 tx=0).stop 2024-09-15T03:59:06.652 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T03:59:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T03:59:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T03:59:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "leader", 2024-09-15T03:59:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 12, 2024-09-15T03:59:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T03:59:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T03:59:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T03:59:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:59:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 0, 2024-09-15T03:59:06.652 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T03:59:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T03:59:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T03:59:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:59:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:59:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:59:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:59:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:59:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:59:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:59:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:59:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:59:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:59:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:59:06.653 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T03:59:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:59:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:59:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:59:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:59:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:59:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:59:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:59:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:59:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:59:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:59:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T03:59:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T03:59:06.654 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T03:59:06.655 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T03:59:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T03:59:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T03:59:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T03:59:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T03:59:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T03:59:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T03:59:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T03:59:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T03:59:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:59:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T03:59:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T03:59:06.656 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T03:59:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T03:59:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:59:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:59:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:59:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T03:59:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:59:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:59:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T03:59:06.657 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:59:06.658 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.658 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:59:06.658 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.658 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T03:59:06.658 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T03:59:06.658 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:59:06.658 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:59:06.658 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:59:06.658 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.658 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.658 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T03:59:06.658 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T03:59:06.658 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:59:06.659 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:59:06.659 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.659 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:59:06.659 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T03:59:06.659 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:59:06.659 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.659 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.659 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:59:06.659 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T03:59:06.659 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:59:06.659 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.659 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:59:06.659 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.660 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T03:59:06.660 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T03:59:06.660 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:59:06.660 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:59:06.660 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:59:06.660 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.660 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.660 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T03:59:06.660 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T03:59:06.660 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T03:59:06.660 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T03:59:06.660 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.661 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T03:59:06.661 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T03:59:06.661 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:59:06.661 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.661 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.661 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T03:59:06.661 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T03:59:06.661 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T03:59:06.661 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.661 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:59:06.661 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.661 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T03:59:06.661 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T03:59:06.662 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T03:59:06.662 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T03:59:06.662 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T03:59:06.662 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.662 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:59:06.662 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.662 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T03:59:06.662 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T03:59:06.662 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.662 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:59:06.662 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:59:06.662 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T03:59:06.662 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.662 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:59:06.663 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T03:59:06.663 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.663 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:59:06.663 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:59:06.663 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 3 2024-09-15T03:59:06.663 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.663 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:59:06.663 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T03:59:06.663 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.663 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:59:06.663 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:59:06.663 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 6 2024-09-15T03:59:06.663 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.664 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T03:59:06.664 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": [ 2024-09-15T03:59:06.664 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T03:59:06.664 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:59:06.664 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T03:59:06.664 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T03:59:06.664 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T03:59:06.664 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T03:59:06.664 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T03:59:06.664 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T03:59:06.664 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T03:59:06.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1376a80640 1 -- 172.21.15.110:0/2965403607 >> v2:172.21.15.177:6812/28201 conn(0x7f135c038700 msgr2=0x7f135c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1376a80640 1 --2- 172.21.15.110:0/2965403607 >> v2:172.21.15.177:6812/28201 conn(0x7f135c038700 0x7f135c03abb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1370119410 tx=0x7f1368006cb0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1376a80640 10 monclient: shutdown 2024-09-15T03:59:06.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1376a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:59:06.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1376a80640 1 -- 172.21.15.110:0/2965403607 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f137006e8d0 msgr2=0x7f137012c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:59:06.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1376a80640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f137006e8d0 0x7f137012c570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f136c00ccd0 tx=0x7f136c0053f0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f137501d640 1 -- 172.21.15.110:0/2965403607 reap_dead start 2024-09-15T03:59:06.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1364ff9640 1 -- 172.21.15.110:0/2965403607 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3894+0+0 (unknown 0 0 0) 0x7f136c04b9b0 con 0x7f137006e8d0 2024-09-15T03:59:06.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1376a80640 1 -- 172.21.15.110:0/2965403607 shutdown_connections 2024-09-15T03:59:06.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1376a80640 1 --2- 172.21.15.110:0/2965403607 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f137006e8d0 0x7f137012c570 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T03:59:06.666 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1376a80640 1 -- 172.21.15.110:0/2965403607 >> 172.21.15.110:0/2965403607 conn(0x7f13700635b0 msgr2=0x7f13700a1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:59:06.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1376a80640 1 -- 172.21.15.110:0/2965403607 shutdown_connections 2024-09-15T03:59:06.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T03:59:06.637+0000 7f1376a80640 1 -- 172.21.15.110:0/2965403607 wait complete. 2024-09-15T03:59:06.667 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:59:06.667 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T03:59:07.178 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29079]: starting. 2024-09-15T03:59:07.179 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29079]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29077 2024-09-15T03:59:07.179 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29079]: shutting down. 2024-09-15T03:59:09.307 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:09.307 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29112]: starting. 2024-09-15T03:59:09.308 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29112]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29077 2024-09-15T03:59:09.308 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29112]: created object 0... 2024-09-15T03:59:09.308 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29112]: created object 25... 2024-09-15T03:59:09.308 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29112]: created object 49... 2024-09-15T03:59:09.308 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29112]: finishing. 2024-09-15T03:59:09.308 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29112]: shutting down. 2024-09-15T03:59:09.346 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:09.346 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29114]: starting. 2024-09-15T03:59:09.346 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29114]: listing objects. 2024-09-15T03:59:09.347 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29114]: listed object 0... 2024-09-15T03:59:09.347 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29114]: listed object 25... 2024-09-15T03:59:09.347 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29114]: saw 50 objects 2024-09-15T03:59:09.347 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29114]: shutting down. 2024-09-15T03:59:10.195 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:10.195 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29113]: starting. 2024-09-15T03:59:10.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29113]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29077 2024-09-15T03:59:10.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29113]: shutting down. 2024-09-15T03:59:10.196 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:10.196 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:10.196 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T03:59:10.198 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T03:59:12.290 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29183]: starting. 2024-09-15T03:59:12.290 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29183]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29182 2024-09-15T03:59:12.290 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29183]: created object 0... 2024-09-15T03:59:12.290 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29183]: created object 25... 2024-09-15T03:59:12.290 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29183]: created object 49... 2024-09-15T03:59:12.290 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29183]: finishing. 2024-09-15T03:59:12.291 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29183]: shutting down. 2024-09-15T03:59:13.213 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29184]: starting. 2024-09-15T03:59:13.213 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29184]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29182 2024-09-15T03:59:13.213 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29184]: shutting down. 2024-09-15T03:59:15.321 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:15.321 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29219]: starting. 2024-09-15T03:59:15.321 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29219]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29182 2024-09-15T03:59:15.321 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29219]: created object 0... 2024-09-15T03:59:15.321 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29219]: created object 25... 2024-09-15T03:59:15.321 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29219]: created object 49... 2024-09-15T03:59:15.321 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29219]: finishing. 2024-09-15T03:59:15.321 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29219]: shutting down. 2024-09-15T03:59:15.360 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:15.360 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29221]: starting. 2024-09-15T03:59:15.360 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29221]: listing objects. 2024-09-15T03:59:15.360 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29221]: listed object 0... 2024-09-15T03:59:15.360 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29221]: listed object 25... 2024-09-15T03:59:15.361 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29221]: saw 50 objects 2024-09-15T03:59:15.361 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29221]: shutting down. 2024-09-15T03:59:16.221 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:16.222 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29220]: starting. 2024-09-15T03:59:16.222 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29220]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29182 2024-09-15T03:59:16.222 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29220]: shutting down. 2024-09-15T03:59:16.222 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:16.222 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:16.222 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T03:59:16.225 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T03:59:18.311 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29290]: starting. 2024-09-15T03:59:18.311 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29290]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29289 2024-09-15T03:59:18.311 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29290]: created object 0... 2024-09-15T03:59:18.311 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29290]: created object 25... 2024-09-15T03:59:18.311 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29290]: created object 49... 2024-09-15T03:59:18.311 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29290]: finishing. 2024-09-15T03:59:18.311 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29290]: shutting down. 2024-09-15T03:59:19.239 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29291]: starting. 2024-09-15T03:59:19.239 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29291]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29289 2024-09-15T03:59:19.240 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29291]: shutting down. 2024-09-15T03:59:21.323 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:21.323 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29326]: starting. 2024-09-15T03:59:21.323 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29326]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29289 2024-09-15T03:59:21.323 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29326]: created object 0... 2024-09-15T03:59:21.323 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29326]: created object 25... 2024-09-15T03:59:21.323 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29326]: created object 49... 2024-09-15T03:59:21.323 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29326]: finishing. 2024-09-15T03:59:21.323 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29326]: shutting down. 2024-09-15T03:59:21.356 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:21.356 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29328]: starting. 2024-09-15T03:59:21.356 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29328]: listing objects. 2024-09-15T03:59:21.357 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29328]: listed object 0... 2024-09-15T03:59:21.357 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29328]: listed object 25... 2024-09-15T03:59:21.357 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29328]: saw 50 objects 2024-09-15T03:59:21.357 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29328]: shutting down. 2024-09-15T03:59:22.245 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:22.245 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29327]: starting. 2024-09-15T03:59:22.245 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29327]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29289 2024-09-15T03:59:22.245 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29327]: shutting down. 2024-09-15T03:59:22.246 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:22.246 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:22.246 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T03:59:22.248 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T03:59:24.349 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29397]: starting. 2024-09-15T03:59:24.349 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29397]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29396 2024-09-15T03:59:24.349 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29397]: created object 0... 2024-09-15T03:59:24.350 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29397]: created object 25... 2024-09-15T03:59:24.350 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29397]: created object 49... 2024-09-15T03:59:24.350 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29397]: finishing. 2024-09-15T03:59:24.350 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29397]: shutting down. 2024-09-15T03:59:25.270 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29398]: starting. 2024-09-15T03:59:25.270 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29398]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29396 2024-09-15T03:59:25.271 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29398]: shutting down. 2024-09-15T03:59:27.373 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:27.373 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29433]: starting. 2024-09-15T03:59:27.373 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29433]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29396 2024-09-15T03:59:27.373 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29433]: created object 0... 2024-09-15T03:59:27.373 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29433]: created object 25... 2024-09-15T03:59:27.373 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29433]: created object 49... 2024-09-15T03:59:27.374 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29433]: finishing. 2024-09-15T03:59:27.374 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29433]: shutting down. 2024-09-15T03:59:27.412 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:27.412 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29435]: starting. 2024-09-15T03:59:27.412 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29435]: listing objects. 2024-09-15T03:59:27.412 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29435]: listed object 0... 2024-09-15T03:59:27.412 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29435]: listed object 25... 2024-09-15T03:59:27.412 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29435]: saw 50 objects 2024-09-15T03:59:27.412 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29435]: shutting down. 2024-09-15T03:59:28.094 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:28.095 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29434]: starting. 2024-09-15T03:59:28.095 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29434]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29396 2024-09-15T03:59:28.095 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29434]: shutting down. 2024-09-15T03:59:28.096 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:28.096 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:28.096 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T03:59:28.098 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T03:59:30.188 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29504]: starting. 2024-09-15T03:59:30.189 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29504]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29503 2024-09-15T03:59:30.189 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29504]: created object 0... 2024-09-15T03:59:30.189 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29504]: created object 25... 2024-09-15T03:59:30.189 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29504]: created object 49... 2024-09-15T03:59:30.189 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29504]: finishing. 2024-09-15T03:59:30.189 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29504]: shutting down. 2024-09-15T03:59:31.106 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29505]: starting. 2024-09-15T03:59:31.106 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29505]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29503 2024-09-15T03:59:31.106 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29505]: shutting down. 2024-09-15T03:59:33.167 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:33.167 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29540]: starting. 2024-09-15T03:59:33.167 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29540]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29503 2024-09-15T03:59:33.167 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29540]: created object 0... 2024-09-15T03:59:33.168 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29540]: created object 25... 2024-09-15T03:59:33.168 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29540]: created object 49... 2024-09-15T03:59:33.168 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29540]: finishing. 2024-09-15T03:59:33.168 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29540]: shutting down. 2024-09-15T03:59:33.205 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:33.205 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29542]: starting. 2024-09-15T03:59:33.205 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29542]: listing objects. 2024-09-15T03:59:33.205 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29542]: listed object 0... 2024-09-15T03:59:33.206 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29542]: listed object 25... 2024-09-15T03:59:33.206 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29542]: saw 50 objects 2024-09-15T03:59:33.206 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29542]: shutting down. 2024-09-15T03:59:34.094 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:34.094 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29541]: starting. 2024-09-15T03:59:34.094 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29541]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29503 2024-09-15T03:59:34.094 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29541]: shutting down. 2024-09-15T03:59:34.095 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:34.095 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:34.095 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T03:59:34.097 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T03:59:36.200 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29611]: starting. 2024-09-15T03:59:36.200 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29611]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29610 2024-09-15T03:59:36.200 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29611]: created object 0... 2024-09-15T03:59:36.200 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29611]: created object 25... 2024-09-15T03:59:36.200 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29611]: created object 49... 2024-09-15T03:59:36.200 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29611]: finishing. 2024-09-15T03:59:36.200 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29611]: shutting down. 2024-09-15T03:59:37.111 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29612]: starting. 2024-09-15T03:59:37.111 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29612]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29610 2024-09-15T03:59:37.111 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29612]: shutting down. 2024-09-15T03:59:39.213 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:39.214 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29647]: starting. 2024-09-15T03:59:39.214 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29647]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29610 2024-09-15T03:59:39.214 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29647]: created object 0... 2024-09-15T03:59:39.214 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29647]: created object 25... 2024-09-15T03:59:39.214 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29647]: created object 49... 2024-09-15T03:59:39.214 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29647]: finishing. 2024-09-15T03:59:39.214 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29647]: shutting down. 2024-09-15T03:59:39.252 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:39.253 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29649]: starting. 2024-09-15T03:59:39.253 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29649]: listing objects. 2024-09-15T03:59:39.253 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29649]: listed object 0... 2024-09-15T03:59:39.253 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29649]: listed object 25... 2024-09-15T03:59:39.253 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29649]: saw 50 objects 2024-09-15T03:59:39.253 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29649]: shutting down. 2024-09-15T03:59:40.134 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:40.134 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29648]: starting. 2024-09-15T03:59:40.134 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29648]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29610 2024-09-15T03:59:40.134 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29648]: shutting down. 2024-09-15T03:59:40.135 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:40.135 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:40.135 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T03:59:40.138 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T03:59:42.252 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29718]: starting. 2024-09-15T03:59:42.253 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29718]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29717 2024-09-15T03:59:42.253 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29718]: created object 0... 2024-09-15T03:59:42.253 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29718]: created object 25... 2024-09-15T03:59:42.253 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29718]: created object 49... 2024-09-15T03:59:42.253 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29718]: finishing. 2024-09-15T03:59:42.253 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29718]: shutting down. 2024-09-15T03:59:43.273 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29719]: starting. 2024-09-15T03:59:43.274 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29719]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29717 2024-09-15T03:59:43.274 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29719]: shutting down. 2024-09-15T03:59:45.385 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:45.385 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29754]: starting. 2024-09-15T03:59:45.385 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29754]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29717 2024-09-15T03:59:45.385 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29754]: created object 0... 2024-09-15T03:59:45.386 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29754]: created object 25... 2024-09-15T03:59:45.386 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29754]: created object 49... 2024-09-15T03:59:45.386 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29754]: finishing. 2024-09-15T03:59:45.386 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29754]: shutting down. 2024-09-15T03:59:45.425 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:45.425 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29756]: starting. 2024-09-15T03:59:45.425 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29756]: listing objects. 2024-09-15T03:59:45.425 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29756]: listed object 0... 2024-09-15T03:59:45.425 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29756]: listed object 25... 2024-09-15T03:59:45.426 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29756]: saw 50 objects 2024-09-15T03:59:45.426 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29756]: shutting down. 2024-09-15T03:59:46.302 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:46.302 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29755]: starting. 2024-09-15T03:59:46.302 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29755]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29717 2024-09-15T03:59:46.302 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29755]: shutting down. 2024-09-15T03:59:46.303 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:46.303 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:46.303 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T03:59:46.305 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T03:59:48.404 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29825]: starting. 2024-09-15T03:59:48.404 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29825]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29824 2024-09-15T03:59:48.404 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29825]: created object 0... 2024-09-15T03:59:48.404 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29825]: created object 25... 2024-09-15T03:59:48.404 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29825]: created object 49... 2024-09-15T03:59:48.404 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29825]: finishing. 2024-09-15T03:59:48.405 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29825]: shutting down. 2024-09-15T03:59:49.319 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29826]: starting. 2024-09-15T03:59:49.319 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29826]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29824 2024-09-15T03:59:49.319 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29826]: shutting down. 2024-09-15T03:59:51.434 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:51.434 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29861]: starting. 2024-09-15T03:59:51.434 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29861]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29824 2024-09-15T03:59:51.434 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29861]: created object 0... 2024-09-15T03:59:51.434 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29861]: created object 25... 2024-09-15T03:59:51.434 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29861]: created object 49... 2024-09-15T03:59:51.435 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29861]: finishing. 2024-09-15T03:59:51.435 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29861]: shutting down. 2024-09-15T03:59:51.472 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:51.472 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29863]: starting. 2024-09-15T03:59:51.472 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29863]: listing objects. 2024-09-15T03:59:51.472 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29863]: listed object 0... 2024-09-15T03:59:51.472 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29863]: listed object 25... 2024-09-15T03:59:51.473 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29863]: saw 50 objects 2024-09-15T03:59:51.473 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29863]: shutting down. 2024-09-15T03:59:52.331 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:52.331 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29862]: starting. 2024-09-15T03:59:52.331 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29862]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29824 2024-09-15T03:59:52.331 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29862]: shutting down. 2024-09-15T03:59:52.332 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:52.332 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:52.332 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T03:59:52.334 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T03:59:54.182 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29932]: starting. 2024-09-15T03:59:54.182 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29932]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29931 2024-09-15T03:59:54.182 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29932]: created object 0... 2024-09-15T03:59:54.182 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29932]: created object 25... 2024-09-15T03:59:54.182 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29932]: created object 49... 2024-09-15T03:59:54.182 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29932]: finishing. 2024-09-15T03:59:54.182 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[29932]: shutting down. 2024-09-15T03:59:55.102 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29933]: starting. 2024-09-15T03:59:55.103 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29933]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29931 2024-09-15T03:59:55.103 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[29933]: shutting down. 2024-09-15T03:59:57.193 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:57.193 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29968]: starting. 2024-09-15T03:59:57.194 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29968]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-29931 2024-09-15T03:59:57.194 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29968]: created object 0... 2024-09-15T03:59:57.194 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29968]: created object 25... 2024-09-15T03:59:57.194 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29968]: created object 49... 2024-09-15T03:59:57.194 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29968]: finishing. 2024-09-15T03:59:57.194 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[29968]: shutting down. 2024-09-15T03:59:57.231 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:57.231 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29970]: starting. 2024-09-15T03:59:57.231 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29970]: listing objects. 2024-09-15T03:59:57.232 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29970]: listed object 0... 2024-09-15T03:59:57.232 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29970]: listed object 25... 2024-09-15T03:59:57.232 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29970]: saw 50 objects 2024-09-15T03:59:57.232 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[29970]: shutting down. 2024-09-15T03:59:58.118 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:58.118 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29969]: starting. 2024-09-15T03:59:58.118 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29969]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-29931 2024-09-15T03:59:58.118 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[29969]: shutting down. 2024-09-15T03:59:58.119 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:58.119 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T03:59:58.119 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T03:59:58.121 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:00:00.213 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30039]: starting. 2024-09-15T04:00:00.213 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30039]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30038 2024-09-15T04:00:00.214 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30039]: created object 0... 2024-09-15T04:00:00.214 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30039]: created object 25... 2024-09-15T04:00:00.214 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30039]: created object 49... 2024-09-15T04:00:00.214 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30039]: finishing. 2024-09-15T04:00:00.214 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30039]: shutting down. 2024-09-15T04:00:01.134 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30040]: starting. 2024-09-15T04:00:01.135 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30040]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30038 2024-09-15T04:00:01.135 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30040]: shutting down. 2024-09-15T04:00:03.201 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:03.201 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30075]: starting. 2024-09-15T04:00:03.201 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30075]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30038 2024-09-15T04:00:03.201 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30075]: created object 0... 2024-09-15T04:00:03.201 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30075]: created object 25... 2024-09-15T04:00:03.201 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30075]: created object 49... 2024-09-15T04:00:03.202 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30075]: finishing. 2024-09-15T04:00:03.202 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30075]: shutting down. 2024-09-15T04:00:03.240 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:03.240 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30077]: starting. 2024-09-15T04:00:03.240 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30077]: listing objects. 2024-09-15T04:00:03.241 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30077]: listed object 0... 2024-09-15T04:00:03.241 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30077]: listed object 25... 2024-09-15T04:00:03.241 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30077]: saw 50 objects 2024-09-15T04:00:03.241 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30077]: shutting down. 2024-09-15T04:00:04.100 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:04.101 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30076]: starting. 2024-09-15T04:00:04.101 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30076]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30038 2024-09-15T04:00:04.101 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30076]: shutting down. 2024-09-15T04:00:04.101 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:04.101 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:04.101 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:00:04.104 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:00:06.209 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30147]: starting. 2024-09-15T04:00:06.209 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30147]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30146 2024-09-15T04:00:06.209 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30147]: created object 0... 2024-09-15T04:00:06.209 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30147]: created object 25... 2024-09-15T04:00:06.209 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30147]: created object 49... 2024-09-15T04:00:06.209 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30147]: finishing. 2024-09-15T04:00:06.209 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30147]: shutting down. 2024-09-15T04:00:07.118 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30148]: starting. 2024-09-15T04:00:07.118 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30148]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30146 2024-09-15T04:00:07.118 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30148]: shutting down. 2024-09-15T04:00:09.207 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:09.207 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30183]: starting. 2024-09-15T04:00:09.207 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30183]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30146 2024-09-15T04:00:09.207 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30183]: created object 0... 2024-09-15T04:00:09.207 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30183]: created object 25... 2024-09-15T04:00:09.207 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30183]: created object 49... 2024-09-15T04:00:09.207 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30183]: finishing. 2024-09-15T04:00:09.207 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30183]: shutting down. 2024-09-15T04:00:09.246 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:09.246 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30185]: starting. 2024-09-15T04:00:09.246 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30185]: listing objects. 2024-09-15T04:00:09.246 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30185]: listed object 0... 2024-09-15T04:00:09.246 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30185]: listed object 25... 2024-09-15T04:00:09.246 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30185]: saw 50 objects 2024-09-15T04:00:09.246 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30185]: shutting down. 2024-09-15T04:00:10.130 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:10.130 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30184]: starting. 2024-09-15T04:00:10.130 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30184]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30146 2024-09-15T04:00:10.130 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30184]: shutting down. 2024-09-15T04:00:10.131 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:10.131 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:10.131 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:00:10.133 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:00:12.229 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30254]: starting. 2024-09-15T04:00:12.230 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30254]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30253 2024-09-15T04:00:12.230 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30254]: created object 0... 2024-09-15T04:00:12.230 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30254]: created object 25... 2024-09-15T04:00:12.230 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30254]: created object 49... 2024-09-15T04:00:12.230 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30254]: finishing. 2024-09-15T04:00:12.230 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30254]: shutting down. 2024-09-15T04:00:13.100 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30255]: starting. 2024-09-15T04:00:13.100 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30255]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30253 2024-09-15T04:00:13.100 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30255]: shutting down. 2024-09-15T04:00:15.259 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:15.259 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30290]: starting. 2024-09-15T04:00:15.259 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30290]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30253 2024-09-15T04:00:15.259 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30290]: created object 0... 2024-09-15T04:00:15.259 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30290]: created object 25... 2024-09-15T04:00:15.259 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30290]: created object 49... 2024-09-15T04:00:15.259 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30290]: finishing. 2024-09-15T04:00:15.259 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30290]: shutting down. 2024-09-15T04:00:15.298 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:15.298 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30292]: starting. 2024-09-15T04:00:15.298 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30292]: listing objects. 2024-09-15T04:00:15.299 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30292]: listed object 0... 2024-09-15T04:00:15.299 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30292]: listed object 25... 2024-09-15T04:00:15.299 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30292]: saw 50 objects 2024-09-15T04:00:15.299 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30292]: shutting down. 2024-09-15T04:00:16.184 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:16.184 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30291]: starting. 2024-09-15T04:00:16.184 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30291]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30253 2024-09-15T04:00:16.184 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30291]: shutting down. 2024-09-15T04:00:16.185 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:16.185 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:16.185 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:00:16.187 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:00:18.310 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30361]: starting. 2024-09-15T04:00:18.310 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30361]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30360 2024-09-15T04:00:18.311 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30361]: created object 0... 2024-09-15T04:00:18.311 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30361]: created object 25... 2024-09-15T04:00:18.311 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30361]: created object 49... 2024-09-15T04:00:18.311 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30361]: finishing. 2024-09-15T04:00:18.311 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30361]: shutting down. 2024-09-15T04:00:19.195 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30362]: starting. 2024-09-15T04:00:19.195 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30362]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30360 2024-09-15T04:00:19.195 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30362]: shutting down. 2024-09-15T04:00:21.306 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:21.306 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30397]: starting. 2024-09-15T04:00:21.306 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30397]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30360 2024-09-15T04:00:21.306 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30397]: created object 0... 2024-09-15T04:00:21.306 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30397]: created object 25... 2024-09-15T04:00:21.306 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30397]: created object 49... 2024-09-15T04:00:21.306 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30397]: finishing. 2024-09-15T04:00:21.306 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30397]: shutting down. 2024-09-15T04:00:21.344 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:21.344 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30399]: starting. 2024-09-15T04:00:21.344 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30399]: listing objects. 2024-09-15T04:00:21.344 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30399]: listed object 0... 2024-09-15T04:00:21.344 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30399]: listed object 25... 2024-09-15T04:00:21.344 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30399]: saw 50 objects 2024-09-15T04:00:21.345 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30399]: shutting down. 2024-09-15T04:00:22.209 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:22.209 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30398]: starting. 2024-09-15T04:00:22.210 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30398]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30360 2024-09-15T04:00:22.210 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30398]: shutting down. 2024-09-15T04:00:22.210 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:22.210 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:22.210 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:00:22.213 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:00:24.220 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30468]: starting. 2024-09-15T04:00:24.220 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30468]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30467 2024-09-15T04:00:24.220 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30468]: created object 0... 2024-09-15T04:00:24.220 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30468]: created object 25... 2024-09-15T04:00:24.220 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30468]: created object 49... 2024-09-15T04:00:24.220 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30468]: finishing. 2024-09-15T04:00:24.220 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30468]: shutting down. 2024-09-15T04:00:25.111 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30469]: starting. 2024-09-15T04:00:25.111 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30469]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30467 2024-09-15T04:00:25.111 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30469]: shutting down. 2024-09-15T04:00:27.233 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:27.233 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30504]: starting. 2024-09-15T04:00:27.233 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30504]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30467 2024-09-15T04:00:27.233 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30504]: created object 0... 2024-09-15T04:00:27.233 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30504]: created object 25... 2024-09-15T04:00:27.233 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30504]: created object 49... 2024-09-15T04:00:27.233 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30504]: finishing. 2024-09-15T04:00:27.233 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30504]: shutting down. 2024-09-15T04:00:28.105 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:28.105 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30506]: starting. 2024-09-15T04:00:28.105 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30506]: listing objects. 2024-09-15T04:00:28.105 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30506]: listed object 0... 2024-09-15T04:00:28.105 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30506]: listed object 25... 2024-09-15T04:00:28.105 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30506]: saw 50 objects 2024-09-15T04:00:28.105 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30506]: shutting down. 2024-09-15T04:00:28.116 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:28.116 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30505]: starting. 2024-09-15T04:00:28.116 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30505]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30467 2024-09-15T04:00:28.116 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30505]: shutting down. 2024-09-15T04:00:28.117 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:28.117 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:28.117 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:00:28.120 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:00:30.226 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30575]: starting. 2024-09-15T04:00:30.227 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30575]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30574 2024-09-15T04:00:30.227 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30575]: created object 0... 2024-09-15T04:00:30.227 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30575]: created object 25... 2024-09-15T04:00:30.227 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30575]: created object 49... 2024-09-15T04:00:30.227 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30575]: finishing. 2024-09-15T04:00:30.227 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30575]: shutting down. 2024-09-15T04:00:31.134 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30576]: starting. 2024-09-15T04:00:31.134 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30576]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30574 2024-09-15T04:00:31.134 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30576]: shutting down. 2024-09-15T04:00:33.216 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:33.216 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30611]: starting. 2024-09-15T04:00:33.216 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30611]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30574 2024-09-15T04:00:33.216 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30611]: created object 0... 2024-09-15T04:00:33.217 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30611]: created object 25... 2024-09-15T04:00:33.217 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30611]: created object 49... 2024-09-15T04:00:33.217 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30611]: finishing. 2024-09-15T04:00:33.217 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30611]: shutting down. 2024-09-15T04:00:33.255 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:33.255 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30613]: starting. 2024-09-15T04:00:33.255 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30613]: listing objects. 2024-09-15T04:00:33.255 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30613]: listed object 0... 2024-09-15T04:00:33.255 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30613]: listed object 25... 2024-09-15T04:00:33.255 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30613]: saw 50 objects 2024-09-15T04:00:33.255 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30613]: shutting down. 2024-09-15T04:00:34.107 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:34.107 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30612]: starting. 2024-09-15T04:00:34.107 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30612]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30574 2024-09-15T04:00:34.107 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30612]: shutting down. 2024-09-15T04:00:34.108 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:34.108 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:34.108 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:00:34.112 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:00:36.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30682]: starting. 2024-09-15T04:00:36.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30682]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30681 2024-09-15T04:00:36.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30682]: created object 0... 2024-09-15T04:00:36.197 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30682]: created object 25... 2024-09-15T04:00:36.197 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30682]: created object 49... 2024-09-15T04:00:36.197 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30682]: finishing. 2024-09-15T04:00:36.197 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30682]: shutting down. 2024-09-15T04:00:36.668 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T04:00:36.668 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T04:00:36.668 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T04:00:36.668 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T04:00:36.671 INFO:tasks.ceph.mon.a:Started 2024-09-15T04:00:36.671 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T04:00:36.672 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:00:36.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.774+0000 7ff58d114640 10 monclient: get_monmap_and_config 2024-09-15T04:00:36.794 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.774+0000 7ff58d114640 10 monclient: build_initial_monmap 2024-09-15T04:00:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.774+0000 7ff58d114640 10 monclient: monmap: 2024-09-15T04:00:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:00:36.775635+0000 2024-09-15T04:00:36.795 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:00:36.775635+0000 2024-09-15T04:00:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:00:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:00:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:00:36.796 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:00:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:00:36.797 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 1 Processor -- start 2024-09-15T04:00:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 1 -- start start 2024-09-15T04:00:36.797 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 10 monclient: init 2024-09-15T04:00:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-15T04:00:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5880a1bf0 0x7ff588131810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 10 monclient(hunting): picked mon.noname-b con 0x7ff5880a1bf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff58809e780 addr v1:172.21.15.177:6789/0 2024-09-15T04:00:36.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff588136a40 0x7ff588138e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:36.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 10 monclient(hunting): picked mon.noname-a con 0x7ff588136a40 addr v2:172.21.15.110:3300/0 2024-09-15T04:00:36.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58809c9b0 con 0x7ff58809e780 2024-09-15T04:00:36.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:36.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5880a0c20 con 0x7ff588136a40 2024-09-15T04:00:36.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58809cc90 con 0x7ff5880a1bf0 2024-09-15T04:00:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58d114640 10 monclient(hunting): authenticate will time out at 860.419785s 2024-09-15T04:00:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58759e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff588136a40 0x7ff588138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff586d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5880a1bf0 0x7ff588131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:36.800 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58759e640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff588136a40 0x7ff588138e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:60974/0 (socket says 172.21.15.110:60974) 2024-09-15T04:00:36.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58759e640 1 -- 172.21.15.110:0/931807433 learned_addr learned my addr 172.21.15.110:0/931807433 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:36.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58759e640 10 monclient(hunting): get_auth_request con 0x7ff588136a40 auth_method 0 2024-09-15T04:00:36.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:36.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:36.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff58759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:36.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff586d9d640 10 monclient(hunting): get_auth_request con 0x7ff5880a1bf0 auth_method 0 2024-09-15T04:00:36.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff586d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:36.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff586d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:36.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.775+0000 7ff586d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:36.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff58759e640 1 -- 172.21.15.110:0/931807433 >> v2:172.21.15.110:3300/0 conn(0x7ff588136a40 msgr2=0x7ff588138e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T04:00:36.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff58759e640 1 -- 172.21.15.110:0/931807433 >> v2:172.21.15.110:3300/0 conn(0x7ff588136a40 msgr2=0x7ff588138e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:36.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff586d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:36.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff586d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:36.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff58759e640 1 --2- 172.21.15.110:0/931807433 >> v2:172.21.15.110:3300/0 conn(0x7ff588136a40 0x7ff588138e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:36.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff586d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:36.803 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff58759e640 1 --2- 172.21.15.110:0/931807433 >> v2:172.21.15.110:3300/0 conn(0x7ff588136a40 0x7ff588138e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:00:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff586d9d640 10 monclient(hunting): handle_auth_done global_id 4757 payload 274 2024-09-15T04:00:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff586d9d640 1 -- 172.21.15.110:0/931807433 >> v2:172.21.15.110:3300/0 conn(0x7ff588136a40 msgr2=0x7ff588138e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:00:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff586d9d640 1 --2- 172.21.15.110:0/931807433 >> v2:172.21.15.110:3300/0 conn(0x7ff588136a40 0x7ff588138e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff586d9d640 1 -- 172.21.15.110:0/931807433 >> v1:172.21.15.177:6789/0 conn(0x7ff58809e780 legacy=0x7ff58809eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff586d9d640 10 monclient: _finish_hunting 0 2024-09-15T04:00:36.804 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff586d9d640 1 monclient: found mon.noname-b 2024-09-15T04:00:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff586d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff586d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:00:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.776+0000 7ff586d9d640 1 -- 172.21.15.110:0/931807433 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5880a1fc0 con 0x7ff5880a1bf0 2024-09-15T04:00:36.805 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff586d9d640 1 -- 172.21.15.110:0/931807433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5880a1bf0 msgr2=0x7ff588131810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:36.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff586d9d640 1 --2- 172.21.15.110:0/931807433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5880a1bf0 0x7ff588131810 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff58809d670 tx=0x7ff5780090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81a45cef9513bc41 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:36.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff585d9b640 1 -- 172.21.15.110:0/931807433 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57802ae50 con 0x7ff5880a1bf0 2024-09-15T04:00:36.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff585d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:36.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff585d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:00:36.806 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff585d9b640 10 monclient: dump: 2024-09-15T04:00:36.806 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:36.807 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:36.807 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:36.807 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:36.807 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:36.807 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:36.808 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:36.808 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:36.808 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:36.808 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:36.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff585d9b640 10 monclient: _finish_auth 0 2024-09-15T04:00:36.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff585d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:36.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff585d9b640 1 -- 172.21.15.110:0/931807433 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff578032440 con 0x7ff5880a1bf0 2024-09-15T04:00:36.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff58d114640 5 monclient: authenticate success, global_id 4757 2024-09-15T04:00:36.809 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff58d114640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:36.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff585d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:36.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff58d114640 10 monclient: get_monmap_and_config success 2024-09-15T04:00:36.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff585d9b640 1 -- 172.21.15.110:0/931807433 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578032820 con 0x7ff5880a1bf0 2024-09-15T04:00:36.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff58d114640 4 set_mon_vals no callback set 2024-09-15T04:00:36.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff585d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:36.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff585d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:36.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff585d9b640 10 monclient: dump: 2024-09-15T04:00:36.811 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:36.811 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:36.811 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:36.811 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:36.812 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:36.812 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:36.812 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:36.812 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:36.812 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:36.813 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:36.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff58d114640 10 monclient: shutdown 2024-09-15T04:00:36.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff58d114640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:36.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff58d114640 1 -- 172.21.15.110:0/931807433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5880a1bf0 msgr2=0x7ff588131810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:36.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff58d114640 1 --2- 172.21.15.110:0/931807433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5880a1bf0 0x7ff588131810 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff58809d670 tx=0x7ff5780090f0 comp rx=0 tx=0).stop 2024-09-15T04:00:36.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff58d114640 1 -- 172.21.15.110:0/931807433 shutdown_connections 2024-09-15T04:00:36.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff58d114640 1 --2- 172.21.15.110:0/931807433 >> v2:172.21.15.110:3300/0 conn(0x7ff588136a40 0x7ff588138e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:36.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff58d114640 1 --2- 172.21.15.110:0/931807433 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff5880a1bf0 0x7ff588131810 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:36.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff58d114640 1 -- 172.21.15.110:0/931807433 >> 172.21.15.110:0/931807433 conn(0x7ff588062dd0 msgr2=0x7ff5880631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:36.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.779+0000 7ff58d114640 1 -- 172.21.15.110:0/931807433 shutdown_connections 2024-09-15T04:00:36.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.780+0000 7ff58d114640 1 -- 172.21.15.110:0/931807433 wait complete. 2024-09-15T04:00:36.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.780+0000 7ff58d114640 10 monclient: build_initial_monmap 2024-09-15T04:00:36.815 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.780+0000 7ff58d114640 10 monclient: monmap: 2024-09-15T04:00:36.815 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:36.815 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:00:36.815 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:00:36.816 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:00:36.816 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:36.816 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:36.816 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:00:36.816 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:00:36.816 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:00:36.817 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:36.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.780+0000 7ff58d114640 1 Processor -- start 2024-09-15T04:00:36.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 1 -- start start 2024-09-15T04:00:36.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 10 monclient: init 2024-09-15T04:00:36.817 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:36.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T04:00:36.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58809e780 0x7ff588141020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:36.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 10 monclient(hunting): picked mon.noname-c con 0x7ff58809e780 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:36.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5880a1bf0 0x7ff58812e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:36.818 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 10 monclient(hunting): picked mon.noname-b con 0x7ff5880a1bf0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:36.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff588136a40 0x7ff588097b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:36.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 10 monclient(hunting): picked mon.noname-a con 0x7ff588136a40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:36.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:36.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff588099c40 con 0x7ff588136a40 2024-09-15T04:00:36.819 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:36.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58809b4d0 con 0x7ff5880a1bf0 2024-09-15T04:00:36.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:36.820 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58809c250 con 0x7ff58809e780 2024-09-15T04:00:36.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:36.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58d114640 10 monclient(hunting): authenticate will time out at 860.425540s 2024-09-15T04:00:36.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58809e780 0x7ff588141020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:36.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58809e780 0x7ff588141020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:59604/0 (socket says 172.21.15.110:59604) 2024-09-15T04:00:36.821 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 1 -- 172.21.15.110:0/3895536390 learned_addr learned my addr 172.21.15.110:0/3895536390 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:36.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 10 monclient(hunting): get_auth_request con 0x7ff58809e780 auth_method 0 2024-09-15T04:00:36.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:36.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:36.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:36.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58759e640 1 --2- 172.21.15.110:0/3895536390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff588136a40 0x7ff588097b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:36.822 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58759e640 10 monclient(hunting): get_auth_request con 0x7ff588136a40 auth_method 0 2024-09-15T04:00:36.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:36.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58759e640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:36.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:36.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:36.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:36.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:36.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58759e640 1 -- 172.21.15.110:0/3895536390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff588136a40 msgr2=0x7ff588097b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T04:00:36.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58759e640 1 -- 172.21.15.110:0/3895536390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff588136a40 msgr2=0x7ff588097b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T04:00:36.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58759e640 1 --2- 172.21.15.110:0/3895536390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff588136a40 0x7ff588097b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:36.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58659c640 1 --2- 172.21.15.110:0/3895536390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5880a1bf0 0x7ff58812e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:36.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58659c640 10 monclient(hunting): get_auth_request con 0x7ff5880a1bf0 auth_method 0 2024-09-15T04:00:36.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:36.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:36.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:36.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58759e640 1 --2- 172.21.15.110:0/3895536390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff588136a40 0x7ff588097b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:00:36.825 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 10 monclient(hunting): handle_auth_done global_id 4760 payload 824 2024-09-15T04:00:36.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 1 -- 172.21.15.110:0/3895536390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5880a1bf0 msgr2=0x7ff58812e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:36.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 1 --2- 172.21.15.110:0/3895536390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5880a1bf0 0x7ff58812e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:36.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 1 -- 172.21.15.110:0/3895536390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff588136a40 msgr2=0x7ff588097b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:00:36.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 1 --2- 172.21.15.110:0/3895536390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff588136a40 0x7ff588097b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:36.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 10 monclient: _finish_hunting 0 2024-09-15T04:00:36.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 1 monclient: found mon.noname-c 2024-09-15T04:00:36.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:36.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:00:36.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff586d9d640 1 -- 172.21.15.110:0/3895536390 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff578009390 con 0x7ff58809e780 2024-09-15T04:00:36.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.781+0000 7ff58659c640 1 --2- 172.21.15.110:0/3895536390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5880a1bf0 0x7ff58812e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T04:00:36.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff586d9d640 1 -- 172.21.15.110:0/3895536390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58809e780 msgr2=0x7ff588141020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:36.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff586d9d640 1 --2- 172.21.15.110:0/3895536390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58809e780 0x7ff588141020 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff578035040 tx=0x7ff578032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:36.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff56effd640 1 -- 172.21.15.110:0/3895536390 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578031990 con 0x7ff58809e780 2024-09-15T04:00:36.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff56effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:36.828 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff56effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:00:36.829 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff56effd640 10 monclient: dump: 2024-09-15T04:00:36.829 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:36.829 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:36.829 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:36.829 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:36.829 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:36.830 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:36.830 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:36.830 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:36.830 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:36.830 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:36.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff56effd640 10 monclient: _finish_auth 0 2024-09-15T04:00:36.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff56effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:36.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff58d114640 5 monclient: authenticate success, global_id 4760 2024-09-15T04:00:36.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff56effd640 1 -- 172.21.15.110:0/3895536390 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff578031af0 con 0x7ff58809e780 2024-09-15T04:00:36.831 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff58d114640 10 monclient: _renew_subs 2024-09-15T04:00:36.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff58d114640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:36.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff58d114640 1 -- 172.21.15.110:0/3895536390 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff58809a300 con 0x7ff58809e780 2024-09-15T04:00:36.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff56effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:36.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff56effd640 1 -- 172.21.15.110:0/3895536390 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff578031e70 con 0x7ff58809e780 2024-09-15T04:00:36.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff56effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:36.832 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff56effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:36.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff56effd640 10 monclient: dump: 2024-09-15T04:00:36.833 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:36.833 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:36.833 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:36.833 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:36.834 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:36.834 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:36.834 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:36.834 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:36.834 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:36.834 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:36.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff58d114640 10 monclient: _renew_subs 2024-09-15T04:00:36.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff58d114640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:36.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff58d114640 1 -- 172.21.15.110:0/3895536390 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff588141560 con 0x7ff58809e780 2024-09-15T04:00:36.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.784+0000 7ff58559a640 4 set_mon_vals no callback set 2024-09-15T04:00:36.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.785+0000 7ff56effd640 1 -- 172.21.15.110:0/3895536390 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7ff578039040 con 0x7ff58809e780 2024-09-15T04:00:36.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.785+0000 7ff58d114640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:36.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.785+0000 7ff58d114640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:36.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.785+0000 7ff58d114640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:36.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.785+0000 7ff58d114640 1 -- 172.21.15.110:0/3895536390 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff588070950 con 0x7ff58809e780 2024-09-15T04:00:36.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.785+0000 7ff56effd640 1 --2- 172.21.15.110:0/3895536390 >> v2:172.21.15.177:6812/28201 conn(0x7ff574038700 0x7ff57403abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:36.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.785+0000 7ff56effd640 1 -- 172.21.15.110:0/3895536390 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 3894+0+0 (secure 0 0 0) 0x7ff57806be90 con 0x7ff58809e780 2024-09-15T04:00:36.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.785+0000 7ff58659c640 1 --2- 172.21.15.110:0/3895536390 >> v2:172.21.15.177:6812/28201 conn(0x7ff574038700 0x7ff57403abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:36.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.785+0000 7ff58659c640 10 monclient: get_auth_request con 0x7ff574038700 auth_method 0 2024-09-15T04:00:36.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.788+0000 7ff58659c640 1 --2- 172.21.15.110:0/3895536390 >> v2:172.21.15.177:6812/28201 conn(0x7ff574038700 0x7ff57403abb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff57c002860 tx=0x7ff57c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:36.837 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.790+0000 7ff56effd640 1 -- 172.21.15.110:0/3895536390 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff578030570 con 0x7ff58809e780 2024-09-15T04:00:36.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.790+0000 7ff56effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:36.838 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:36.790+0000 7ff56effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:00:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.004+0000 7ff58d114640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:00:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.004+0000 7ff58d114640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:00:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.004+0000 7ff58d114640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.004+0000 7ff58d114640 1 -- 172.21.15.110:0/3895536390 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff5880a26c0 con 0x7ff58809e780 2024-09-15T04:00:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.004+0000 7ff56effd640 1 -- 172.21.15.110:0/3895536390 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7ff57803f070 con 0x7ff58809e780 2024-09-15T04:00:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.004+0000 7ff56effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:00:37.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.004+0000 7ff56effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:00:37.006 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":12,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":90,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:00:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.006+0000 7ff58d114640 1 -- 172.21.15.110:0/3895536390 >> v2:172.21.15.177:6812/28201 conn(0x7ff574038700 msgr2=0x7ff57403abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.006+0000 7ff58d114640 1 --2- 172.21.15.110:0/3895536390 >> v2:172.21.15.177:6812/28201 conn(0x7ff574038700 0x7ff57403abb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff57c002860 tx=0x7ff57c009040 comp rx=0 tx=0).stop 2024-09-15T04:00:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.006+0000 7ff58d114640 10 monclient: shutdown 2024-09-15T04:00:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.006+0000 7ff58d114640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.006+0000 7ff58d114640 1 -- 172.21.15.110:0/3895536390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58809e780 msgr2=0x7ff588141020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.006+0000 7ff58d114640 1 --2- 172.21.15.110:0/3895536390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58809e780 0x7ff588141020 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff578035040 tx=0x7ff578032000 comp rx=0 tx=0).stop 2024-09-15T04:00:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.006+0000 7ff58d114640 1 -- 172.21.15.110:0/3895536390 shutdown_connections 2024-09-15T04:00:37.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.006+0000 7ff58d114640 1 --2- 172.21.15.110:0/3895536390 >> v2:172.21.15.177:6812/28201 conn(0x7ff574038700 0x7ff57403abb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.006+0000 7ff58d114640 1 --2- 172.21.15.110:0/3895536390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff588136a40 0x7ff588097b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.006+0000 7ff58d114640 1 --2- 172.21.15.110:0/3895536390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff5880a1bf0 0x7ff58812e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.006+0000 7ff58d114640 1 --2- 172.21.15.110:0/3895536390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff58809e780 0x7ff588141020 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.006+0000 7ff58d114640 1 -- 172.21.15.110:0/3895536390 >> 172.21.15.110:0/3895536390 conn(0x7ff588062dd0 msgr2=0x7ff5880a4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.006+0000 7ff58d114640 1 -- 172.21.15.110:0/3895536390 shutdown_connections 2024-09-15T04:00:37.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:37.006+0000 7ff58d114640 1 -- 172.21.15.110:0/3895536390 wait complete. 2024-09-15T04:00:37.123 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30683]: starting. 2024-09-15T04:00:37.123 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30683]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30681 2024-09-15T04:00:37.123 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30683]: shutting down. 2024-09-15T04:00:40.017 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:00:40.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.119+0000 7f8aceb9d640 10 monclient: get_monmap_and_config 2024-09-15T04:00:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.119+0000 7f8aceb9d640 10 monclient: build_initial_monmap 2024-09-15T04:00:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.119+0000 7f8aceb9d640 10 monclient: monmap: 2024-09-15T04:00:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:00:40.120936+0000 2024-09-15T04:00:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:00:40.120936+0000 2024-09-15T04:00:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:00:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:00:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:00:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:00:40.121 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:00:40.122 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.120+0000 7f8aceb9d640 1 Processor -- start 2024-09-15T04:00:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.120+0000 7f8aceb9d640 1 -- start start 2024-09-15T04:00:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.120+0000 7f8aceb9d640 10 monclient: init 2024-09-15T04:00:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8aceb9d640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8aceb9d640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-15T04:00:40.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8aceb9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ac8131630 addr v1:172.21.15.110:6789/0 2024-09-15T04:00:40.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8aceb9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:40.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8aceb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ac806e5a0 addr v2:172.21.15.110:3300/0 2024-09-15T04:00:40.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8aceb9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 0x7f8ac8138970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:40.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8aceb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ac806eeb0 addr v2:172.21.15.177:3300/0 2024-09-15T04:00:40.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8aceb9d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac8068390 con 0x7f8ac8131630 2024-09-15T04:00:40.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8aceb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:40.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8aceb9d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac8069570 con 0x7f8ac806e5a0 2024-09-15T04:00:40.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8aceb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:40.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8aceb9d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac8069850 con 0x7f8ac806eeb0 2024-09-15T04:00:40.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8aceb9d640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:40.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8aceb9d640 10 monclient(hunting): authenticate will time out at 863.765279s 2024-09-15T04:00:40.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8ac7fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:40.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8acc939640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8ac8131630 0x7f8ac8131a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:54776/0 (socket says 172.21.15.110:54776) 2024-09-15T04:00:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8ac7fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39888/0 (socket says 172.21.15.110:39888) 2024-09-15T04:00:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 learned_addr learned my addr 172.21.15.110:0/3792885530 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:00:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request con 0x7f8ac806e5a0 auth_method 0 2024-09-15T04:00:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8ac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8ac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 msgr2=0x7f8ac806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:00:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 msgr2=0x7f8ac806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:00:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8acd13a640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 0x7f8ac8138970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8acd13a640 10 monclient(hunting): get_auth_request con 0x7f8ac806eeb0 auth_method 0 2024-09-15T04:00:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8acd13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:40.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8acd13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8acd13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8acd13a640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 msgr2=0x7f8ac8138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T04:00:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8acd13a640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 msgr2=0x7f8ac8138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8acd13a640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 0x7f8ac8138970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.121+0000 7f8acd13a640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 0x7f8ac8138970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:00:40.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.123+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 >> v1:172.21.15.110:6789/0 conn(0x7f8ac8131630 legacy=0x7f8ac8131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:40.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:40.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request con 0x7f8ac806e5a0 auth_method 0 2024-09-15T04:00:40.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:40.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8ac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:40.323 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8ac7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:40.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 msgr2=0x7f8ac806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:00:40.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 msgr2=0x7f8ac806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:40.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:40.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T04:00:40.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8acd13a640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 0x7f8ac8138970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:40.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8acd13a640 10 monclient(hunting): get_auth_request con 0x7f8ac806eeb0 auth_method 0 2024-09-15T04:00:40.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8acd13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:40.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8acd13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:40.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8acd13a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:40.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8acd13a640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 msgr2=0x7f8ac8138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T04:00:40.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8acd13a640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 msgr2=0x7f8ac8138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:40.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8acd13a640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 0x7f8ac8138970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:40.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.322+0000 7f8acd13a640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 0x7f8ac8138970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T04:00:40.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:40.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request con 0x7f8ac806e5a0 auth_method 0 2024-09-15T04:00:40.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:40.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8ac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:40.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8ac7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:40.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 msgr2=0x7f8ac806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:00:40.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 msgr2=0x7f8ac806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:40.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:40.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8acd13a640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 0x7f8ac8138970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:40.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T04:00:40.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8acd13a640 10 monclient(hunting): get_auth_request con 0x7f8ac806eeb0 auth_method 0 2024-09-15T04:00:40.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8acd13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:40.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8acd13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:40.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8acd13a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:40.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8acd13a640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 msgr2=0x7f8ac8138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T04:00:40.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8acd13a640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 msgr2=0x7f8ac8138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:40.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8acd13a640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 0x7f8ac8138970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:40.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:40.723+0000 7f8acd13a640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 0x7f8ac8138970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T04:00:41.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:41.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request con 0x7f8ac806e5a0 auth_method 0 2024-09-15T04:00:41.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:41.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8ac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:41.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8ac7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:41.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 msgr2=0x7f8ac806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:00:41.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 msgr2=0x7f8ac806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:41.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:41.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T04:00:41.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8acd13a640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 0x7f8ac8138970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:41.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8acd13a640 10 monclient(hunting): get_auth_request con 0x7f8ac806eeb0 auth_method 0 2024-09-15T04:00:41.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8acd13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:41.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8acd13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:41.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8acd13a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:41.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8acd13a640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 msgr2=0x7f8ac8138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T04:00:41.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8acd13a640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 msgr2=0x7f8ac8138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:41.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8acd13a640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 0x7f8ac8138970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:41.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:41.524+0000 7f8acd13a640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 0x7f8ac8138970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T04:00:43.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 10 monclient(hunting): tick 2024-09-15T04:00:43.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 1 monclient(hunting): continuing hunt 2024-09-15T04:00:43.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T04:00:43.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 msgr2=0x7f8ac8138970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:00:43.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8ac806eeb0 0x7f8ac8138970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:43.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 msgr2=0x7f8ac806e970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:00:43.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.110:3300/0 conn(0x7f8ac806e5a0 0x7f8ac806e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:43.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 1 -- 172.21.15.110:0/3792885530 >> v1:172.21.15.110:6789/0 conn(0x7f8ac8131630 legacy=0x7f8ac8131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:43.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 10 monclient: _add_conns ranks=[2,1] 2024-09-15T04:00:43.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 0x7f8aac005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:43.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 10 monclient(hunting): picked mon.noname-b con 0x7f8aac0036d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:43.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8aac006240 addr v1:172.21.15.177:6789/0 2024-09-15T04:00:43.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 1 -- 172.21.15.110:0/3792885530 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ac8132110 con 0x7f8aac006240 2024-09-15T04:00:43.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:43.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 1 -- 172.21.15.110:0/3792885530 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac8069570 con 0x7f8aac0036d0 2024-09-15T04:00:43.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8ac67fc640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:43.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 0x7f8aac005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:43.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8acc939640 10 monclient(hunting): get_auth_request con 0x7f8aac0036d0 auth_method 0 2024-09-15T04:00:43.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8acc939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:43.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8acc939640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:43.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8acc939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:43.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T04:00:43.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:43.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 0x7f8aac005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:43.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.122+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 0x7f8aac005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:00:43.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.124+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v1:172.21.15.177:6789/0 conn(0x7f8aac006240 legacy=0x7f8aac0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:43.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.323+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 0x7f8aac005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:43.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.323+0000 7f8acc939640 10 monclient(hunting): get_auth_request con 0x7f8aac0036d0 auth_method 0 2024-09-15T04:00:43.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.323+0000 7f8acc939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:43.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.323+0000 7f8acc939640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:43.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.323+0000 7f8acc939640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:43.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.323+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T04:00:43.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.323+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:43.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.323+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 0x7f8aac005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:43.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.323+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 0x7f8aac005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T04:00:43.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.724+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 0x7f8aac005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:43.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.724+0000 7f8acc939640 10 monclient(hunting): get_auth_request con 0x7f8aac0036d0 auth_method 0 2024-09-15T04:00:43.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.724+0000 7f8acc939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:43.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.724+0000 7f8acc939640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:43.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.724+0000 7f8acc939640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:43.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.724+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T04:00:43.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.724+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:43.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.724+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 0x7f8aac005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:43.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:43.724+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 0x7f8aac005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T04:00:44.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:44.525+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 0x7f8aac005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:44.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:44.525+0000 7f8acc939640 10 monclient(hunting): get_auth_request con 0x7f8aac0036d0 auth_method 0 2024-09-15T04:00:44.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:44.525+0000 7f8acc939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:44.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:44.525+0000 7f8acc939640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:44.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:44.525+0000 7f8acc939640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:44.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:44.525+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T04:00:44.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:44.525+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:44.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:44.525+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 0x7f8aac005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:44.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:44.525+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 0x7f8aac005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T04:00:46.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 10 monclient(hunting): tick 2024-09-15T04:00:46.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 1 monclient(hunting): continuing hunt 2024-09-15T04:00:46.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T04:00:46.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 1 -- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:00:46.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8aac0036d0 0x7f8aac005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:46.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 1 -- 172.21.15.110:0/3792885530 >> v1:172.21.15.177:6789/0 conn(0x7f8aac006240 legacy=0x7f8aac0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:46.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-15T04:00:46.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8acd13a640 1 -- 172.21.15.110:0/3792885530 reap_dead start 2024-09-15T04:00:46.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ac8131630 addr v1:172.21.15.110:6789/0 2024-09-15T04:00:46.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 0x7f8aac005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:46.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 10 monclient(hunting): picked mon.noname-c con 0x7f8aac00a840 addr v2:172.21.15.177:3300/0 2024-09-15T04:00:46.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8aac008ea0 addr v1:172.21.15.177:6789/0 2024-09-15T04:00:46.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 1 -- 172.21.15.110:0/3792885530 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aac00b010 con 0x7f8ac8131630 2024-09-15T04:00:46.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 1 -- 172.21.15.110:0/3792885530 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aac009340 con 0x7f8aac008ea0 2024-09-15T04:00:46.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:46.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 1 -- 172.21.15.110:0/3792885530 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ac8069570 con 0x7f8aac00a840 2024-09-15T04:00:46.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac67fc640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:46.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 0x7f8aac005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:46.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request con 0x7f8aac00a840 auth_method 0 2024-09-15T04:00:46.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:46.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:46.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:46.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:00:46.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:46.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 0x7f8aac005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:46.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.121+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 0x7f8aac005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:00:46.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.123+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 >> v1:172.21.15.110:6789/0 conn(0x7f8ac8131630 legacy=0x7f8aac00a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:46.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.123+0000 7f8acd13a640 1 -- 172.21.15.110:0/3792885530 >> v1:172.21.15.177:6789/0 conn(0x7f8aac008ea0 legacy=0x7f8aac0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:46.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.322+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 0x7f8aac005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:46.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.322+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request con 0x7f8aac00a840 auth_method 0 2024-09-15T04:00:46.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.322+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:46.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.322+0000 7f8ac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:46.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.322+0000 7f8ac7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:46.324 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.322+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:00:46.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.322+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:46.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.322+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 0x7f8aac005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:46.325 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.322+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 0x7f8aac005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T04:00:46.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.723+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 0x7f8aac005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:46.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.723+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request con 0x7f8aac00a840 auth_method 0 2024-09-15T04:00:46.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.723+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:46.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.723+0000 7f8ac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:46.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.723+0000 7f8ac7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:46.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.723+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:00:46.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.723+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:46.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.723+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 0x7f8aac005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:46.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:46.723+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 0x7f8aac005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T04:00:47.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:47.525+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 0x7f8aac005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:47.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:47.525+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request con 0x7f8aac00a840 auth_method 0 2024-09-15T04:00:47.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:47.525+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:47.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:47.525+0000 7f8ac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:47.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:47.525+0000 7f8ac7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:47.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:47.525+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:00:47.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:47.525+0000 7f8ac7fff640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:00:47.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:47.525+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 0x7f8aac005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:47.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:47.525+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 0x7f8aac005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T04:00:48.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 -- 172.21.15.110:0/3792885530 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002283655 0 0) 0x7f8aac009340 con 0x7f8aac008ea0 2024-09-15T04:00:48.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:48.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:48.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient(hunting): my global_id is 4765 2024-09-15T04:00:48.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 -- 172.21.15.110:0/3792885530 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ab80034e0 con 0x7f8aac008ea0 2024-09-15T04:00:48.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 -- 172.21.15.110:0/3792885530 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3945335858 0 0) 0x7f8ab80034e0 con 0x7f8aac008ea0 2024-09-15T04:00:48.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 -- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 msgr2=0x7f8aac005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:00:48.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 --2- 172.21.15.110:0/3792885530 >> v2:172.21.15.177:3300/0 conn(0x7f8aac00a840 0x7f8aac005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:48.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 -- 172.21.15.110:0/3792885530 >> v1:172.21.15.110:6789/0 conn(0x7f8ac8131630 legacy=0x7f8aac00a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:48.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient: _finish_hunting 0 2024-09-15T04:00:48.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 monclient: found mon.noname-c-legacy 2024-09-15T04:00:48.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:48.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.177:6789/0 2024-09-15T04:00:48.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 -- 172.21.15.110:0/3792885530 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac009520 con 0x7f8aac008ea0 2024-09-15T04:00:48.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient: _finish_auth 0 2024-09-15T04:00:48.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:48.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 -- 172.21.15.110:0/3792885530 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3165669920 0 0) 0x7f8ac0009a90 con 0x7f8aac008ea0 2024-09-15T04:00:48.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:48.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T04:00:48.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient: dump: 2024-09-15T04:00:48.207 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:48.207 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:48.207 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:48.208 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:48.208 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:48.208 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:48.208 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:48.208 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:48.208 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:48.208 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:48.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] but i'm connected to v1:172.21.15.177:6789/0, reconnecting 2024-09-15T04:00:48.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:48.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 -- 172.21.15.110:0/3792885530 >> v1:172.21.15.177:6789/0 conn(0x7f8aac008ea0 legacy=0x7f8aac0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:48.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T04:00:48.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ab80039d0 0x7f8ab8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:48.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient(hunting): picked mon.a con 0x7f8ab80039d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:48.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ab8004780 0x7f8ab8004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:48.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient(hunting): picked mon.b con 0x7f8ab8004780 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:48.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ab8005250 0x7f8ab800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:48.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient(hunting): picked mon.c con 0x7f8ab8005250 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:48.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:48.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 -- 172.21.15.110:0/3792885530 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac8069570 con 0x7f8ab80039d0 2024-09-15T04:00:48.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:48.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 -- 172.21.15.110:0/3792885530 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab800c1d0 con 0x7f8ab8004780 2024-09-15T04:00:48.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:48.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 1 -- 172.21.15.110:0/3792885530 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab800c390 con 0x7f8ab8005250 2024-09-15T04:00:48.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:48.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac77fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T04:00:48.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ab80039d0 0x7f8ab8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:48.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8acc939640 10 monclient(hunting): get_auth_request con 0x7f8ab80039d0 auth_method 0 2024-09-15T04:00:48.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8acc939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:48.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8acc939640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:48.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8acc939640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:48.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8acd13a640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ab8005250 0x7f8ab800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:48.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ab8004780 0x7f8ab8004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:48.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8acd13a640 10 monclient(hunting): get_auth_request con 0x7f8ab8005250 auth_method 0 2024-09-15T04:00:48.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8acd13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:48.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8acd13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:48.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8acd13a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:48.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request con 0x7f8ab8004780 auth_method 0 2024-09-15T04:00:48.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.202+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:48.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8ac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:48.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8ac7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:48.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:48.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:48.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:00:48.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8ac7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:48.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8ac7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:48.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8ac7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:00:48.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acd13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:48.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acd13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:48.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acd13a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:00:48.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 10 monclient(hunting): handle_auth_done global_id 4765 payload 293 2024-09-15T04:00:48.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ab8005250 msgr2=0x7f8ab800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:48.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ab8005250 0x7f8ab800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:48.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ab8004780 msgr2=0x7f8ab8004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:48.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ab8004780 0x7f8ab8004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:48.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 10 monclient: _finish_hunting 0 2024-09-15T04:00:48.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 1 monclient: found mon.a 2024-09-15T04:00:48.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:48.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:48.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab80034e0 con 0x7f8ab80039d0 2024-09-15T04:00:48.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 10 monclient: _finish_auth 0 2024-09-15T04:00:48.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acc939640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:48.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8aceb9d640 5 monclient: authenticate success, global_id 4765 2024-09-15T04:00:48.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8aceb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:48.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ab8004780 0x7f8ab8004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:00:48.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acd13a640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ab8005250 0x7f8ab800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:00:48.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.203+0000 7f8acd13a640 1 -- 172.21.15.110:0/3792885530 reap_dead start 2024-09-15T04:00:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8acc939640 1 -- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ab80039d0 msgr2=0x7f8ab8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8acc939640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ab80039d0 0x7f8ab8003de0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8abc007050 tx=0x7f8abc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f2c76bd63b5550 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8ac77fe640 1 -- 172.21.15.110:0/3792885530 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc02b860 con 0x7f8ab80039d0 2024-09-15T04:00:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8ac77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8ac77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8ac77fe640 10 monclient: dump: 2024-09-15T04:00:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:48.219 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8ac77fe640 1 -- 172.21.15.110:0/3792885530 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8abc02f070 con 0x7f8ab80039d0 2024-09-15T04:00:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8aceb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8ac77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8aceb9d640 10 monclient: get_monmap_and_config success 2024-09-15T04:00:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8aceb9d640 4 set_mon_vals no callback set 2024-09-15T04:00:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8ac77fe640 1 -- 172.21.15.110:0/3792885530 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc02b9c0 con 0x7f8ab80039d0 2024-09-15T04:00:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8ac77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8ac77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.206+0000 7f8ac77fe640 10 monclient: dump: 2024-09-15T04:00:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:48.220 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.207+0000 7f8aceb9d640 10 monclient: shutdown 2024-09-15T04:00:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.207+0000 7f8aceb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.207+0000 7f8aceb9d640 1 -- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ab80039d0 msgr2=0x7f8ab8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.207+0000 7f8aceb9d640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ab80039d0 0x7f8ab8003de0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8abc007050 tx=0x7f8abc009040 comp rx=0 tx=0).stop 2024-09-15T04:00:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.207+0000 7f8aceb9d640 1 -- 172.21.15.110:0/3792885530 shutdown_connections 2024-09-15T04:00:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.207+0000 7f8aceb9d640 1 --2- 172.21.15.110:0/3792885530 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ab80039d0 0x7f8ab8003de0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.207+0000 7f8aceb9d640 1 -- 172.21.15.110:0/3792885530 >> 172.21.15.110:0/3792885530 conn(0x7f8ac80639f0 msgr2=0x7f8ac80a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.207+0000 7f8aceb9d640 1 -- 172.21.15.110:0/3792885530 shutdown_connections 2024-09-15T04:00:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.207+0000 7f8aceb9d640 1 -- 172.21.15.110:0/3792885530 wait complete. 2024-09-15T04:00:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.208+0000 7f8aceb9d640 10 monclient: build_initial_monmap 2024-09-15T04:00:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.208+0000 7f8aceb9d640 10 monclient: monmap: 2024-09-15T04:00:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:00:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:00:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:00:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:00:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:00:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:00:48.222 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.208+0000 7f8aceb9d640 1 Processor -- start 2024-09-15T04:00:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 1 -- start start 2024-09-15T04:00:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 10 monclient: init 2024-09-15T04:00:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T04:00:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ab80039d0 0x7f8ac809bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ab80039d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ac812e7f0 0x7f8ac812ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ac812e7f0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ac81191c0 0x7f8ac8119630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:48.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ac81191c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac80688c0 con 0x7f8ac81191c0 2024-09-15T04:00:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac8130f70 con 0x7f8ac812e7f0 2024-09-15T04:00:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac8140c50 con 0x7f8ab80039d0 2024-09-15T04:00:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.209+0000 7f8aceb9d640 10 monclient(hunting): authenticate will time out at 871.853909s 2024-09-15T04:00:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ab80039d0 0x7f8ac809bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acd13a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ac81191c0 0x7f8ac8119630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acd13a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ac81191c0 0x7f8ac8119630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39942/0 (socket says 172.21.15.110:39942) 2024-09-15T04:00:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ab80039d0 0x7f8ac809bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:53380/0 (socket says 172.21.15.110:53380) 2024-09-15T04:00:48.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acd13a640 1 -- 172.21.15.110:0/1310930390 learned_addr learned my addr 172.21.15.110:0/1310930390 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acd13a640 10 monclient(hunting): get_auth_request con 0x7f8ac81191c0 auth_method 0 2024-09-15T04:00:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acd13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acd13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acd13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 10 monclient(hunting): get_auth_request con 0x7f8ab80039d0 auth_method 0 2024-09-15T04:00:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/1310930390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ac812e7f0 0x7f8ac812ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request con 0x7f8ac812e7f0 auth_method 0 2024-09-15T04:00:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8ac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8ac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8ac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:48.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acd13a640 1 -- 172.21.15.110:0/1310930390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ac81191c0 msgr2=0x7f8ac8119630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T04:00:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acd13a640 1 -- 172.21.15.110:0/1310930390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ac81191c0 msgr2=0x7f8ac8119630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T04:00:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acd13a640 1 --2- 172.21.15.110:0/1310930390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ac81191c0 0x7f8ac8119630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acd13a640 1 --2- 172.21.15.110:0/1310930390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ac81191c0 0x7f8ac8119630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:00:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8ac7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8ac7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8ac7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 10 monclient(hunting): handle_auth_done global_id 4775 payload 824 2024-09-15T04:00:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 1 -- 172.21.15.110:0/1310930390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ac812e7f0 msgr2=0x7f8ac812ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 1 --2- 172.21.15.110:0/1310930390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ac812e7f0 0x7f8ac812ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:48.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 1 -- 172.21.15.110:0/1310930390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ac81191c0 msgr2=0x7f8ac8119630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:00:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 1 --2- 172.21.15.110:0/1310930390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ac81191c0 0x7f8ac8119630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 10 monclient: _finish_hunting 0 2024-09-15T04:00:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 1 monclient: found mon.noname-c 2024-09-15T04:00:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:00:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.210+0000 7f8acc939640 1 -- 172.21.15.110:0/1310930390 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac811a360 con 0x7f8ab80039d0 2024-09-15T04:00:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.212+0000 7f8acc939640 1 -- 172.21.15.110:0/1310930390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ab80039d0 msgr2=0x7f8ac809bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.212+0000 7f8acc939640 1 --2- 172.21.15.110:0/1310930390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ab80039d0 0x7f8ac809bf60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8abc033000 tx=0x7f8abc037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.212+0000 7f8ac4ff9640 1 -- 172.21.15.110:0/1310930390 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc02b860 con 0x7f8ab80039d0 2024-09-15T04:00:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.212+0000 7f8ac4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8ac4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:00:48.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8ac4ff9640 10 monclient: dump: 2024-09-15T04:00:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:48.227 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8ac4ff9640 10 monclient: _finish_auth 0 2024-09-15T04:00:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8ac4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8ac4ff9640 1 -- 172.21.15.110:0/1310930390 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8abc005050 con 0x7f8ab80039d0 2024-09-15T04:00:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8aceb9d640 5 monclient: authenticate success, global_id 4775 2024-09-15T04:00:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8aceb9d640 10 monclient: _renew_subs 2024-09-15T04:00:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8aceb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8aceb9d640 1 -- 172.21.15.110:0/1310930390 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac811aa20 con 0x7f8ab80039d0 2024-09-15T04:00:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8ac4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8ac4ff9640 1 -- 172.21.15.110:0/1310930390 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc02ba00 con 0x7f8ab80039d0 2024-09-15T04:00:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8ac4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8ac4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8ac4ff9640 10 monclient: dump: 2024-09-15T04:00:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:48.229 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8ac6ffd640 4 set_mon_vals no callback set 2024-09-15T04:00:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8aceb9d640 10 monclient: _renew_subs 2024-09-15T04:00:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8aceb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.213+0000 7f8aceb9d640 1 -- 172.21.15.110:0/1310930390 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac81415b0 con 0x7f8ab80039d0 2024-09-15T04:00:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.214+0000 7f8ac4ff9640 1 -- 172.21.15.110:0/1310930390 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f8abc0047a0 con 0x7f8ab80039d0 2024-09-15T04:00:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.214+0000 7f8aceb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.214+0000 7f8aceb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:48.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.214+0000 7f8aceb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:48.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.214+0000 7f8aceb9d640 1 -- 172.21.15.110:0/1310930390 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac8070950 con 0x7f8ab80039d0 2024-09-15T04:00:48.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.214+0000 7f8ac4ff9640 1 --2- 172.21.15.110:0/1310930390 >> v2:172.21.15.177:6812/28201 conn(0x7f8aac039820 0x7f8aac03bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:48.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.214+0000 7f8ac4ff9640 1 -- 172.21.15.110:0/1310930390 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f8abc06bc20 con 0x7f8ab80039d0 2024-09-15T04:00:48.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.214+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/1310930390 >> v2:172.21.15.177:6812/28201 conn(0x7f8aac039820 0x7f8aac03bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:48.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.214+0000 7f8ac7fff640 10 monclient: get_auth_request con 0x7f8aac039820 auth_method 0 2024-09-15T04:00:48.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.217+0000 7f8ac7fff640 1 --2- 172.21.15.110:0/1310930390 >> v2:172.21.15.177:6812/28201 conn(0x7f8aac039820 0x7f8aac03bcd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4006e70 tx=0x7f8ab4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:48.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.218+0000 7f8ac4ff9640 1 -- 172.21.15.110:0/1310930390 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8abc03eb10 con 0x7f8ab80039d0 2024-09-15T04:00:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.218+0000 7f8ac4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.218+0000 7f8ac4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:00:48.352 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:48.352 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30718]: starting. 2024-09-15T04:00:48.352 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30718]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30681 2024-09-15T04:00:48.352 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30718]: created object 0... 2024-09-15T04:00:48.353 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30718]: created object 25... 2024-09-15T04:00:48.353 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30718]: created object 49... 2024-09-15T04:00:48.353 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30718]: finishing. 2024-09-15T04:00:48.353 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30718]: shutting down. 2024-09-15T04:00:48.394 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:48.395 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30720]: starting. 2024-09-15T04:00:48.395 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30720]: listing objects. 2024-09-15T04:00:48.395 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30720]: listed object 0... 2024-09-15T04:00:48.395 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30720]: listed object 25... 2024-09-15T04:00:48.395 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30720]: saw 50 objects 2024-09-15T04:00:48.395 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30720]: shutting down. 2024-09-15T04:00:48.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.455+0000 7f8aceb9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:00:48.456 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.455+0000 7f8aceb9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:00:48.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.455+0000 7f8aceb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:48.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.455+0000 7f8aceb9d640 1 -- 172.21.15.110:0/1310930390 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8ac8070950 con 0x7f8ab80039d0 2024-09-15T04:00:48.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.455+0000 7f8ac4ff9640 1 -- 172.21.15.110:0/1310930390 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f8abc03eb10 con 0x7f8ab80039d0 2024-09-15T04:00:48.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.455+0000 7f8ac4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:00:48.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.455+0000 7f8ac4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:00:48.457 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":18,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:00:48.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.457+0000 7f8aceb9d640 1 -- 172.21.15.110:0/1310930390 >> v2:172.21.15.177:6812/28201 conn(0x7f8aac039820 msgr2=0x7f8aac03bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:48.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.457+0000 7f8aceb9d640 1 --2- 172.21.15.110:0/1310930390 >> v2:172.21.15.177:6812/28201 conn(0x7f8aac039820 0x7f8aac03bcd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4006e70 tx=0x7f8ab4008040 comp rx=0 tx=0).stop 2024-09-15T04:00:48.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.457+0000 7f8aceb9d640 10 monclient: shutdown 2024-09-15T04:00:48.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.457+0000 7f8aceb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:48.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.457+0000 7f8aceb9d640 1 -- 172.21.15.110:0/1310930390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ab80039d0 msgr2=0x7f8ac809bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:48.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.457+0000 7f8aceb9d640 1 --2- 172.21.15.110:0/1310930390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ab80039d0 0x7f8ac809bf60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8abc033000 tx=0x7f8abc037000 comp rx=0 tx=0).stop 2024-09-15T04:00:48.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.457+0000 7f8aceb9d640 1 -- 172.21.15.110:0/1310930390 shutdown_connections 2024-09-15T04:00:48.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.457+0000 7f8aceb9d640 1 --2- 172.21.15.110:0/1310930390 >> v2:172.21.15.177:6812/28201 conn(0x7f8aac039820 0x7f8aac03bcd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:48.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.457+0000 7f8aceb9d640 1 --2- 172.21.15.110:0/1310930390 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ac81191c0 0x7f8ac8119630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:48.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.457+0000 7f8aceb9d640 1 --2- 172.21.15.110:0/1310930390 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ac812e7f0 0x7f8ac812ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:48.459 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.457+0000 7f8aceb9d640 1 --2- 172.21.15.110:0/1310930390 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ab80039d0 0x7f8ac809bf60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:48.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.458+0000 7f8aceb9d640 1 -- 172.21.15.110:0/1310930390 >> 172.21.15.110:0/1310930390 conn(0x7f8ac80639f0 msgr2=0x7f8ac806c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:48.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.458+0000 7f8aceb9d640 1 -- 172.21.15.110:0/1310930390 shutdown_connections 2024-09-15T04:00:48.460 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:48.458+0000 7f8aceb9d640 1 -- 172.21.15.110:0/1310930390 wait complete. 2024-09-15T04:00:49.266 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:49.266 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30719]: starting. 2024-09-15T04:00:49.266 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30719]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30681 2024-09-15T04:00:49.266 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30719]: shutting down. 2024-09-15T04:00:49.267 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:49.267 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:00:49.267 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:00:49.270 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:00:50.869 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30789]: starting. 2024-09-15T04:00:50.870 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30789]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30788 2024-09-15T04:00:50.870 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30789]: created object 0... 2024-09-15T04:00:50.870 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30789]: created object 25... 2024-09-15T04:00:50.870 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30789]: created object 49... 2024-09-15T04:00:50.870 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30789]: finishing. 2024-09-15T04:00:50.870 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30789]: shutting down. 2024-09-15T04:00:51.470 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:00:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.572+0000 7f2e41d1d640 10 monclient: get_monmap_and_config 2024-09-15T04:00:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.572+0000 7f2e41d1d640 10 monclient: build_initial_monmap 2024-09-15T04:00:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.572+0000 7f2e41d1d640 10 monclient: monmap: 2024-09-15T04:00:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:00:51.573337+0000 2024-09-15T04:00:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:00:51.573337+0000 2024-09-15T04:00:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:00:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:00:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:00:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:00:51.574 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:00:51.574 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:51.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 1 Processor -- start 2024-09-15T04:00:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 1 -- start start 2024-09-15T04:00:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 10 monclient: init 2024-09-15T04:00:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T04:00:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e3c067130 addr v1:172.21.15.110:6789/0 2024-09-15T04:00:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 0x7f2e3c0678d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f2e3c067500 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f2e3c134920 0x7f2e3c136d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f2e3c134920 addr v2:172.21.15.177:3300/0 2024-09-15T04:00:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e3c09c670 con 0x7f2e3c067130 2024-09-15T04:00:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:51.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e3c09cd70 con 0x7f2e3c134920 2024-09-15T04:00:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c09ca90 con 0x7f2e3c067500 2024-09-15T04:00:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e41d1d640 10 monclient(hunting): authenticate will time out at 875.217728s 2024-09-15T04:00:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e3b7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2e3c067130 0x7f2e3c09dd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51166/0 (socket says 172.21.15.110:51166) 2024-09-15T04:00:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e3b7fe640 1 -- 172.21.15.110:0/4060533540 learned_addr learned my addr 172.21.15.110:0/4060533540 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:00:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e3affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 0x7f2e3c0678d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e3affd640 10 monclient(hunting): get_auth_request con 0x7f2e3c067500 auth_method 0 2024-09-15T04:00:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e3affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:51.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.573+0000 7f2e3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.574+0000 7f2e3bfff640 1 --2- 172.21.15.110:0/4060533540 >> v2:172.21.15.177:3300/0 conn(0x7f2e3c134920 0x7f2e3c136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.574+0000 7f2e3bfff640 10 monclient(hunting): get_auth_request con 0x7f2e3c134920 auth_method 0 2024-09-15T04:00:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.574+0000 7f2e3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.574+0000 7f2e3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.574+0000 7f2e3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.574+0000 7f2e3affd640 10 monclient(hunting): handle_auth_done global_id 4787 payload 274 2024-09-15T04:00:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.574+0000 7f2e3affd640 1 -- 172.21.15.110:0/4060533540 >> v2:172.21.15.177:3300/0 conn(0x7f2e3c134920 msgr2=0x7f2e3c136d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.574+0000 7f2e3affd640 1 --2- 172.21.15.110:0/4060533540 >> v2:172.21.15.177:3300/0 conn(0x7f2e3c134920 0x7f2e3c136d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.574+0000 7f2e3affd640 1 -- 172.21.15.110:0/4060533540 >> v1:172.21.15.110:6789/0 conn(0x7f2e3c067130 legacy=0x7f2e3c09dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.574+0000 7f2e3affd640 10 monclient: _finish_hunting 0 2024-09-15T04:00:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.574+0000 7f2e3affd640 1 monclient: found mon.noname-b 2024-09-15T04:00:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.574+0000 7f2e3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.574+0000 7f2e3affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:00:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.574+0000 7f2e3affd640 1 -- 172.21.15.110:0/4060533540 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c09d050 con 0x7f2e3c067500 2024-09-15T04:00:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.576+0000 7f2e3affd640 1 -- 172.21.15.110:0/4060533540 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 msgr2=0x7f2e3c0678d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.576+0000 7f2e3affd640 1 --2- 172.21.15.110:0/4060533540 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 0x7f2e3c0678d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c06e110 tx=0x7f2e240090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86605b142677e529 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e3a7fc640 1 -- 172.21.15.110:0/4060533540 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2402adb0 con 0x7f2e3c067500 2024-09-15T04:00:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e3a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:00:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e3a7fc640 10 monclient: dump: 2024-09-15T04:00:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:51.587 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e3a7fc640 10 monclient: _finish_auth 0 2024-09-15T04:00:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e41d1d640 5 monclient: authenticate success, global_id 4787 2024-09-15T04:00:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e3a7fc640 1 -- 172.21.15.110:0/4060533540 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e24032440 con 0x7f2e3c067500 2024-09-15T04:00:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e41d1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e41d1d640 10 monclient: get_monmap_and_config success 2024-09-15T04:00:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e41d1d640 4 set_mon_vals no callback set 2024-09-15T04:00:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e3a7fc640 1 -- 172.21.15.110:0/4060533540 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e240327e0 con 0x7f2e3c067500 2024-09-15T04:00:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e3a7fc640 10 monclient: dump: 2024-09-15T04:00:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:51.588 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e41d1d640 10 monclient: shutdown 2024-09-15T04:00:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e41d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:51.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e41d1d640 1 -- 172.21.15.110:0/4060533540 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 msgr2=0x7f2e3c0678d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e41d1d640 1 --2- 172.21.15.110:0/4060533540 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 0x7f2e3c0678d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c06e110 tx=0x7f2e240090f0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e41d1d640 1 -- 172.21.15.110:0/4060533540 shutdown_connections 2024-09-15T04:00:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e41d1d640 1 --2- 172.21.15.110:0/4060533540 >> v2:172.21.15.177:3300/0 conn(0x7f2e3c134920 0x7f2e3c136d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e41d1d640 1 --2- 172.21.15.110:0/4060533540 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 0x7f2e3c0678d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e41d1d640 1 -- 172.21.15.110:0/4060533540 >> 172.21.15.110:0/4060533540 conn(0x7f2e3c09e5e0 msgr2=0x7f2e3c09e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e41d1d640 1 -- 172.21.15.110:0/4060533540 shutdown_connections 2024-09-15T04:00:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.577+0000 7f2e41d1d640 1 -- 172.21.15.110:0/4060533540 wait complete. 2024-09-15T04:00:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.578+0000 7f2e41d1d640 10 monclient: build_initial_monmap 2024-09-15T04:00:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.578+0000 7f2e41d1d640 10 monclient: monmap: 2024-09-15T04:00:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:00:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:00:51.589 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:00:51.590 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:51.590 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:51.590 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:00:51.590 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:00:51.590 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:00:51.590 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:51.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.578+0000 7f2e41d1d640 1 Processor -- start 2024-09-15T04:00:51.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.578+0000 7f2e41d1d640 1 -- start start 2024-09-15T04:00:51.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.578+0000 7f2e41d1d640 10 monclient: init 2024-09-15T04:00:51.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:51.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T04:00:51.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e3c067130 0x7f2e3c099380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:51.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 10 monclient(hunting): picked mon.noname-b con 0x7f2e3c067130 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 0x7f2e3c0998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 10 monclient(hunting): picked mon.noname-c con 0x7f2e3c067500 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e3c134920 0x7f2e3c099e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 10 monclient(hunting): picked mon.noname-a con 0x7f2e3c134920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c130090 con 0x7f2e3c134920 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c09ba40 con 0x7f2e3c067130 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3c1306f0 con 0x7f2e3c067500 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e41d1d640 10 monclient(hunting): authenticate will time out at 875.223181s 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e3c134920 0x7f2e3c099e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 0x7f2e3c0998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:51.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e3c134920 0x7f2e3c099e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53918/0 (socket says 172.21.15.110:53918) 2024-09-15T04:00:51.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 0x7f2e3c0998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50684/0 (socket says 172.21.15.110:50684) 2024-09-15T04:00:51.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3bfff640 1 -- 172.21.15.110:0/1472644840 learned_addr learned my addr 172.21.15.110:0/1472644840 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:51.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3bfff640 10 monclient(hunting): get_auth_request con 0x7f2e3c134920 auth_method 0 2024-09-15T04:00:51.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:51.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:51.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:51.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 10 monclient(hunting): get_auth_request con 0x7f2e3c067500 auth_method 0 2024-09-15T04:00:51.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:51.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:51.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:51.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3b7fe640 1 --2- 172.21.15.110:0/1472644840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e3c067130 0x7f2e3c099380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:51.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3b7fe640 10 monclient(hunting): get_auth_request con 0x7f2e3c067130 auth_method 0 2024-09-15T04:00:51.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 10 monclient(hunting): handle_auth_done global_id 4790 payload 824 2024-09-15T04:00:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 1 -- 172.21.15.110:0/1472644840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e3c067130 msgr2=0x7f2e3c099380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:51.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 1 --2- 172.21.15.110:0/1472644840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e3c067130 0x7f2e3c099380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 1 -- 172.21.15.110:0/1472644840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e3c134920 msgr2=0x7f2e3c099e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 1 --2- 172.21.15.110:0/1472644840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e3c134920 0x7f2e3c099e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 10 monclient: _finish_hunting 0 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 1 monclient: found mon.noname-c 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3affd640 1 -- 172.21.15.110:0/1472644840 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e24009390 con 0x7f2e3c067500 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.579+0000 7f2e3bfff640 1 --2- 172.21.15.110:0/1472644840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e3c134920 0x7f2e3c099e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.581+0000 7f2e3affd640 1 -- 172.21.15.110:0/1472644840 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 msgr2=0x7f2e3c0998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.581+0000 7f2e3affd640 1 --2- 172.21.15.110:0/1472644840 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 0x7f2e3c0998c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2e24031000 tx=0x7f2e24032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e337fe640 1 -- 172.21.15.110:0/1472644840 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24031d40 con 0x7f2e3c067500 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e337fe640 10 monclient: dump: 2024-09-15T04:00:51.594 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:51.595 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:51.595 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.595 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.595 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:51.595 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:51.595 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:51.595 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:51.595 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:51.595 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:51.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e337fe640 10 monclient: _finish_auth 0 2024-09-15T04:00:51.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:51.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e337fe640 1 -- 172.21.15.110:0/1472644840 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2403d440 con 0x7f2e3c067500 2024-09-15T04:00:51.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e41d1d640 5 monclient: authenticate success, global_id 4790 2024-09-15T04:00:51.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e337fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e41d1d640 10 monclient: _renew_subs 2024-09-15T04:00:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e337fe640 1 -- 172.21.15.110:0/1472644840 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24031ea0 con 0x7f2e3c067500 2024-09-15T04:00:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e41d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e41d1d640 1 -- 172.21.15.110:0/1472644840 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3c136870 con 0x7f2e3c067500 2024-09-15T04:00:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e337fe640 10 monclient: dump: 2024-09-15T04:00:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:51.596 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e39ffb640 4 set_mon_vals no callback set 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e41d1d640 10 monclient: _renew_subs 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e41d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e41d1d640 1 -- 172.21.15.110:0/1472644840 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3c1195c0 con 0x7f2e3c067500 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.582+0000 7f2e337fe640 1 -- 172.21.15.110:0/1472644840 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f2e24039040 con 0x7f2e3c067500 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.583+0000 7f2e41d1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.583+0000 7f2e41d1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.583+0000 7f2e41d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.583+0000 7f2e337fe640 1 --2- 172.21.15.110:0/1472644840 >> v2:172.21.15.177:6812/28201 conn(0x7f2e1c038780 0x7f2e1c03ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.583+0000 7f2e41d1d640 1 -- 172.21.15.110:0/1472644840 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3c070950 con 0x7f2e3c067500 2024-09-15T04:00:51.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.583+0000 7f2e337fe640 1 -- 172.21.15.110:0/1472644840 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f2e2406b140 con 0x7f2e3c067500 2024-09-15T04:00:51.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.583+0000 7f2e3b7fe640 1 --2- 172.21.15.110:0/1472644840 >> v2:172.21.15.177:6812/28201 conn(0x7f2e1c038780 0x7f2e1c03ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:51.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.583+0000 7f2e3b7fe640 10 monclient: get_auth_request con 0x7f2e1c038780 auth_method 0 2024-09-15T04:00:51.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.586+0000 7f2e3b7fe640 1 --2- 172.21.15.110:0/1472644840 >> v2:172.21.15.177:6812/28201 conn(0x7f2e1c038780 0x7f2e1c03ac30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c007fd0 tx=0x7f2e2c007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:51.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.587+0000 7f2e337fe640 1 -- 172.21.15.110:0/1472644840 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f2e24099e40 con 0x7f2e3c067500 2024-09-15T04:00:51.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.587+0000 7f2e337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:51.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.587+0000 7f2e337fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:00:51.769 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30790]: starting. 2024-09-15T04:00:51.769 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30790]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30788 2024-09-15T04:00:51.769 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30790]: shutting down. 2024-09-15T04:00:51.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.809+0000 7f2e41d1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:00:51.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.809+0000 7f2e41d1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:00:51.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.809+0000 7f2e41d1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:51.810 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.809+0000 7f2e41d1d640 1 -- 172.21.15.110:0/1472644840 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2e3c0a2950 con 0x7f2e3c067500 2024-09-15T04:00:51.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.809+0000 7f2e337fe640 1 -- 172.21.15.110:0/1472644840 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f2e24037020 con 0x7f2e3c067500 2024-09-15T04:00:51.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.809+0000 7f2e337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:00:51.811 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.809+0000 7f2e337fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:00:51.811 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:00:51.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.811+0000 7f2e41d1d640 1 -- 172.21.15.110:0/1472644840 >> v2:172.21.15.177:6812/28201 conn(0x7f2e1c038780 msgr2=0x7f2e1c03ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:51.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.811+0000 7f2e41d1d640 1 --2- 172.21.15.110:0/1472644840 >> v2:172.21.15.177:6812/28201 conn(0x7f2e1c038780 0x7f2e1c03ac30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c007fd0 tx=0x7f2e2c007f00 comp rx=0 tx=0).stop 2024-09-15T04:00:51.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.811+0000 7f2e41d1d640 10 monclient: shutdown 2024-09-15T04:00:51.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.811+0000 7f2e41d1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:51.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.811+0000 7f2e41d1d640 1 -- 172.21.15.110:0/1472644840 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 msgr2=0x7f2e3c0998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:51.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.811+0000 7f2e41d1d640 1 --2- 172.21.15.110:0/1472644840 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 0x7f2e3c0998c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2e24031000 tx=0x7f2e24032000 comp rx=0 tx=0).stop 2024-09-15T04:00:51.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.811+0000 7f2e41d1d640 1 -- 172.21.15.110:0/1472644840 shutdown_connections 2024-09-15T04:00:51.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.811+0000 7f2e41d1d640 1 --2- 172.21.15.110:0/1472644840 >> v2:172.21.15.177:6812/28201 conn(0x7f2e1c038780 0x7f2e1c03ac30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.811+0000 7f2e41d1d640 1 --2- 172.21.15.110:0/1472644840 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2e3c134920 0x7f2e3c099e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.811+0000 7f2e41d1d640 1 --2- 172.21.15.110:0/1472644840 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f2e3c067500 0x7f2e3c0998c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.811+0000 7f2e41d1d640 1 --2- 172.21.15.110:0/1472644840 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2e3c067130 0x7f2e3c099380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.813 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.812+0000 7f2e41d1d640 1 -- 172.21.15.110:0/1472644840 >> 172.21.15.110:0/1472644840 conn(0x7f2e3c09e5e0 msgr2=0x7f2e3c136b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:51.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.812+0000 7f2e41d1d640 1 -- 172.21.15.110:0/1472644840 shutdown_connections 2024-09-15T04:00:51.814 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.812+0000 7f2e41d1d640 1 -- 172.21.15.110:0/1472644840 wait complete. 2024-09-15T04:00:51.823 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T04:00:51.823 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T04:00:51.823 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T04:00:51.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.915+0000 7f54abc34640 10 monclient: get_monmap_and_config 2024-09-15T04:00:51.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.915+0000 7f54abc34640 10 monclient: build_initial_monmap 2024-09-15T04:00:51.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.915+0000 7f54abc34640 10 monclient: monmap: 2024-09-15T04:00:51.919 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:51.919 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:51.919 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:00:51.916240+0000 2024-09-15T04:00:51.919 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:00:51.916240+0000 2024-09-15T04:00:51.919 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:51.919 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:51.919 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:00:51.919 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:00:51.919 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:00:51.919 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:00:51.919 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:00:51.919 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:51.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.915+0000 7f54abc34640 1 Processor -- start 2024-09-15T04:00:51.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 1 -- start start 2024-09-15T04:00:51.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 10 monclient: init 2024-09-15T04:00:51.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:51.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T04:00:51.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f54a40a1cf0 0x7f54a409e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:51.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 10 monclient(hunting): picked mon.noname-a con 0x7f54a40a1cf0 addr v2:172.21.15.110:3300/0 2024-09-15T04:00:51.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54a4131a80 addr v1:172.21.15.110:6789/0 2024-09-15T04:00:51.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54a4136940 0x7f54a4138d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:51.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 10 monclient(hunting): picked mon.noname-b con 0x7f54a4136940 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:51.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a409cb70 con 0x7f54a4131a80 2024-09-15T04:00:51.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:51.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a40a20c0 con 0x7f54a40a1cf0 2024-09-15T04:00:51.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:51.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a409c330 con 0x7f54a4136940 2024-09-15T04:00:51.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:51.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54abc34640 10 monclient(hunting): authenticate will time out at 875.560516s 2024-09-15T04:00:51.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54a4136940 0x7f54a4138d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:51.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54a99d0640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f54a40a1cf0 0x7f54a409e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:51.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54a4136940 0x7f54a4138d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50686/0 (socket says 172.21.15.110:50686) 2024-09-15T04:00:51.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 1 -- 172.21.15.110:0/3370798442 learned_addr learned my addr 172.21.15.110:0/3370798442 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:51.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 10 monclient(hunting): get_auth_request con 0x7f54a4136940 auth_method 0 2024-09-15T04:00:51.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:51.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:51.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:51.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54a99d0640 10 monclient(hunting): get_auth_request con 0x7f54a40a1cf0 auth_method 0 2024-09-15T04:00:51.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54a99d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:51.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54a99d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:51.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54a99d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:51.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54a99d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:51.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54a99d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:51.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54a99d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:51.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:51.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:51.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:51.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 10 monclient(hunting): handle_auth_done global_id 4811 payload 274 2024-09-15T04:00:51.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 1 -- 172.21.15.110:0/3370798442 >> v2:172.21.15.110:3300/0 conn(0x7f54a40a1cf0 msgr2=0x7f54a409e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:51.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 1 --2- 172.21.15.110:0/3370798442 >> v2:172.21.15.110:3300/0 conn(0x7f54a40a1cf0 0x7f54a409e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 1 -- 172.21.15.110:0/3370798442 >> v1:172.21.15.110:6789/0 conn(0x7f54a4131a80 legacy=0x7f54a409eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:51.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 10 monclient: _finish_hunting 0 2024-09-15T04:00:51.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 1 monclient: found mon.noname-b 2024-09-15T04:00:51.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:51.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:00:51.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.916+0000 7f54aa1d1640 1 -- 172.21.15.110:0/3370798442 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a4131fb0 con 0x7f54a4136940 2024-09-15T04:00:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54aa1d1640 1 -- 172.21.15.110:0/3370798442 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54a4136940 msgr2=0x7f54a4138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54aa1d1640 1 --2- 172.21.15.110:0/3370798442 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54a4136940 0x7f54a4138d20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f549c009f20 tx=0x7f549c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f6b5bd5a50886bf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54a89ce640 1 -- 172.21.15.110:0/3370798442 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c007610 con 0x7f54a4136940 2024-09-15T04:00:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54a89ce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54a89ce640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:00:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54a89ce640 10 monclient: dump: 2024-09-15T04:00:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:51.929 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54a89ce640 10 monclient: _finish_auth 0 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54a89ce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54a89ce640 1 -- 172.21.15.110:0/3370798442 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549c02d050 con 0x7f54a4136940 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54abc34640 5 monclient: authenticate success, global_id 4811 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54abc34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54a89ce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54a89ce640 1 -- 172.21.15.110:0/3370798442 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c0079e0 con 0x7f54a4136940 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54abc34640 10 monclient: get_monmap_and_config success 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54abc34640 4 set_mon_vals no callback set 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54a89ce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54a89ce640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.919+0000 7f54a89ce640 10 monclient: dump: 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:51.931 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.920+0000 7f54abc34640 10 monclient: shutdown 2024-09-15T04:00:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.920+0000 7f54abc34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.920+0000 7f54abc34640 1 -- 172.21.15.110:0/3370798442 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54a4136940 msgr2=0x7f54a4138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.920+0000 7f54abc34640 1 --2- 172.21.15.110:0/3370798442 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54a4136940 0x7f54a4138d20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f549c009f20 tx=0x7f549c009840 comp rx=0 tx=0).stop 2024-09-15T04:00:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.920+0000 7f54abc34640 1 -- 172.21.15.110:0/3370798442 shutdown_connections 2024-09-15T04:00:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.920+0000 7f54abc34640 1 --2- 172.21.15.110:0/3370798442 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54a4136940 0x7f54a4138d20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.920+0000 7f54abc34640 1 --2- 172.21.15.110:0/3370798442 >> v2:172.21.15.110:3300/0 conn(0x7f54a40a1cf0 0x7f54a409e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.920+0000 7f54abc34640 1 -- 172.21.15.110:0/3370798442 >> 172.21.15.110:0/3370798442 conn(0x7f54a4062f70 msgr2=0x7f54a4063340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.920+0000 7f54abc34640 1 -- 172.21.15.110:0/3370798442 shutdown_connections 2024-09-15T04:00:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.920+0000 7f54abc34640 1 -- 172.21.15.110:0/3370798442 wait complete. 2024-09-15T04:00:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 10 monclient: build_initial_monmap 2024-09-15T04:00:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 10 monclient: monmap: 2024-09-15T04:00:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:00:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:00:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:00:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:00:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:00:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:00:51.933 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 1 Processor -- start 2024-09-15T04:00:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 1 -- start start 2024-09-15T04:00:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 10 monclient: init 2024-09-15T04:00:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T04:00:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54a40a1cf0 0x7f54a409b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 10 monclient(hunting): picked mon.noname-b con 0x7f54a40a1cf0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54a4131a80 0x7f54a4097060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 10 monclient(hunting): picked mon.noname-c con 0x7f54a4131a80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4136940 0x7f54a40975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 10 monclient(hunting): picked mon.noname-a con 0x7f54a4136940 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a409b880 con 0x7f54a4136940 2024-09-15T04:00:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a409acb0 con 0x7f54a40a1cf0 2024-09-15T04:00:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a40a11e0 con 0x7f54a4131a80 2024-09-15T04:00:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.921+0000 7f54abc34640 10 monclient(hunting): authenticate will time out at 875.565991s 2024-09-15T04:00:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54a91cf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54a4131a80 0x7f54a4097060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4136940 0x7f54a40975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4136940 0x7f54a40975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53930/0 (socket says 172.21.15.110:53930) 2024-09-15T04:00:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54a91cf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54a4131a80 0x7f54a4097060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50690/0 (socket says 172.21.15.110:50690) 2024-09-15T04:00:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 1 -- 172.21.15.110:0/3291424069 learned_addr learned my addr 172.21.15.110:0/3291424069 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 10 monclient(hunting): get_auth_request con 0x7f54a4136940 auth_method 0 2024-09-15T04:00:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54a91cf640 10 monclient(hunting): get_auth_request con 0x7f54a4131a80 auth_method 0 2024-09-15T04:00:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54a91cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54a91cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54a91cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54a91cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54a91cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54a91cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 10 monclient(hunting): handle_auth_done global_id 54120 payload 824 2024-09-15T04:00:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 1 -- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54a4131a80 msgr2=0x7f54a4097060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54a4131a80 0x7f54a4097060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 1 -- 172.21.15.110:0/3291424069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54a40a1cf0 msgr2=0x7f54a409b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54a99d0640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54a40a1cf0 0x7f54a409b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54a40a1cf0 0x7f54a409b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 10 monclient: _finish_hunting 0 2024-09-15T04:00:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 1 monclient: found mon.noname-a 2024-09-15T04:00:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:00:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54aa1d1640 1 -- 172.21.15.110:0/3291424069 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549c009ae0 con 0x7f54a4136940 2024-09-15T04:00:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54a99d0640 10 monclient: get_auth_request con 0x7f54a40a1cf0 auth_method 0 2024-09-15T04:00:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54a91cf640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f54a4131a80 0x7f54a4097060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:00:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.922+0000 7f54a99d0640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54a40a1cf0 0x7f54a409b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:00:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f54aa1d1640 1 -- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4136940 msgr2=0x7f54a40975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f54aa1d1640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4136940 0x7f54a40975a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f549c009f80 tx=0x7f549c036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f5495ffb640 1 -- 172.21.15.110:0/3291424069 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c007170 con 0x7f54a4136940 2024-09-15T04:00:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f5495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f5495ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:00:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f5495ffb640 10 monclient: dump: 2024-09-15T04:00:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:51.938 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f5495ffb640 10 monclient: _finish_auth 0 2024-09-15T04:00:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f5495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f54abc34640 5 monclient: authenticate success, global_id 54120 2024-09-15T04:00:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f5495ffb640 1 -- 172.21.15.110:0/3291424069 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549c032cc0 con 0x7f54a4136940 2024-09-15T04:00:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f54abc34640 10 monclient: _renew_subs 2024-09-15T04:00:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f54abc34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f54abc34640 1 -- 172.21.15.110:0/3291424069 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a4099060 con 0x7f54a4136940 2024-09-15T04:00:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f5495ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f5495ffb640 1 -- 172.21.15.110:0/3291424069 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c03b680 con 0x7f54a4136940 2024-09-15T04:00:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f5495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.925+0000 7f5495ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.925+0000 7f5495ffb640 10 monclient: dump: 2024-09-15T04:00:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:51.939 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.924+0000 7f5497fff640 4 set_mon_vals no callback set 2024-09-15T04:00:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.925+0000 7f54abc34640 10 monclient: _renew_subs 2024-09-15T04:00:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.925+0000 7f54abc34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.925+0000 7f54abc34640 1 -- 172.21.15.110:0/3291424069 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a409a680 con 0x7f54a4136940 2024-09-15T04:00:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.925+0000 7f54abc34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.925+0000 7f54abc34640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4099780 0x7f54a4097ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.925+0000 7f54abc34640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.925+0000 7f54abc34640 1 -- 172.21.15.110:0/3291424069 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a406f1e0 con 0x7f54a4099780 2024-09-15T04:00:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f54a99d0640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4099780 0x7f54a4097ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f54a99d0640 10 monclient: get_auth_request con 0x7f54a4099780 auth_method 0 2024-09-15T04:00:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f54a99d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f54a99d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f54a99d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f5495ffb640 1 -- 172.21.15.110:0/3291424069 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f549c038050 con 0x7f54a4136940 2024-09-15T04:00:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f54a99d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f54a99d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f54a99d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f54a99d0640 10 monclient(hunting): handle_auth_done global_id 54123 payload 824 2024-09-15T04:00:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f54a99d0640 1 -- 172.21.15.110:0/3291424069 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54a40647e0 con 0x7f54a4099780 2024-09-15T04:00:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f5495ffb640 1 --2- 172.21.15.110:0/3291424069 >> v2:172.21.15.177:6812/28201 conn(0x7f5488038700 0x7f548803abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f5495ffb640 1 -- 172.21.15.110:0/3291424069 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f549c02ae40 con 0x7f54a4136940 2024-09-15T04:00:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f54a91cf640 1 --2- 172.21.15.110:0/3291424069 >> v2:172.21.15.177:6812/28201 conn(0x7f5488038700 0x7f548803abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.926+0000 7f54a91cf640 10 monclient: get_auth_request con 0x7f5488038700 auth_method 0 2024-09-15T04:00:51.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.928+0000 7f54a99d0640 1 -- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4099780 msgr2=0x7f54a4097ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:51.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.928+0000 7f54a99d0640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4099780 0x7f54a4097ae0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f54980107d0 tx=0x7f5498008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:51.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.928+0000 7f5495ffb640 1 -- 172.21.15.110:0/3291424069 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498010ea0 con 0x7f54a4099780 2024-09-15T04:00:51.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.928+0000 7f5495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:51.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.928+0000 7f5495ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:51.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.928+0000 7f5495ffb640 10 monclient: dump: 2024-09-15T04:00:51.944 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:51.944 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:51.944 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.944 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.944 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:51.944 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:51.944 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:51.944 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:51.944 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:51.944 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.928+0000 7f5495ffb640 1 -- 172.21.15.110:0/3291424069 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f549800ecf0 con 0x7f54a4099780 2024-09-15T04:00:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.928+0000 7f5495ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.928+0000 7f5495ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:00:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.928+0000 7f5495ffb640 1 -- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4099780 msgr2=0x7f54a4097ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.928+0000 7f5495ffb640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4099780 0x7f54a4097ae0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f54980107d0 tx=0x7f5498008040 comp rx=0 tx=0).stop 2024-09-15T04:00:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.928+0000 7f5495ffb640 1 -- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4099780 msgr2=0x7f54a4097ae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:00:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.928+0000 7f5495ffb640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4099780 0x7f54a4097ae0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.929+0000 7f54a91cf640 1 --2- 172.21.15.110:0/3291424069 >> v2:172.21.15.177:6812/28201 conn(0x7f5488038700 0x7f548803abb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f54a409c5a0 tx=0x7f54a0007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.934+0000 7f54abc34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:00:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.934+0000 7f54abc34640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4134620 0x7f54a4134a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.934+0000 7f54abc34640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.934+0000 7f54abc34640 1 -- 172.21.15.110:0/3291424069 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a4067130 con 0x7f54a4134620 2024-09-15T04:00:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.934+0000 7f54a99d0640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4134620 0x7f54a4134a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.934+0000 7f54a99d0640 10 monclient: get_auth_request con 0x7f54a4134620 auth_method 0 2024-09-15T04:00:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.934+0000 7f54a99d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.934+0000 7f54a99d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.934+0000 7f54a99d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.934+0000 7f54a99d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.934+0000 7f54a99d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.934+0000 7f54a99d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.934+0000 7f54a99d0640 10 monclient(hunting): handle_auth_done global_id 54126 payload 824 2024-09-15T04:00:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.934+0000 7f54a99d0640 1 -- 172.21.15.110:0/3291424069 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f54a4134f60 con 0x7f54a4134620 2024-09-15T04:00:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f54a99d0640 1 -- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4134620 msgr2=0x7f54a4134a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f54a99d0640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4134620 0x7f54a4134a20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f549801a000 tx=0x7f549801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f5495ffb640 1 -- 172.21.15.110:0/3291424069 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498021070 con 0x7f54a4134620 2024-09-15T04:00:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f5495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f5495ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f5495ffb640 10 monclient: dump: 2024-09-15T04:00:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:51.947 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f5495ffb640 1 -- 172.21.15.110:0/3291424069 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7f5498005e70 con 0x7f54a4134620 2024-09-15T04:00:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f5495ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:00:51.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f5495ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:00:51.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f5495ffb640 1 -- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4134620 msgr2=0x7f54a4134a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:51.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f5495ffb640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4134620 0x7f54a4134a20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f549801a000 tx=0x7f549801b000 comp rx=0 tx=0).stop 2024-09-15T04:00:51.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f5495ffb640 1 -- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4134620 msgr2=0x7f54a4134a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:00:51.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f5495ffb640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4134620 0x7f54a4134a20 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f549801a000 tx=0x7f549801b000 comp rx=0 tx=0).stop 2024-09-15T04:00:51.948 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:00:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:00:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:00:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "leader", 2024-09-15T04:00:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 22, 2024-09-15T04:00:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:00:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:00:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:00:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:00:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 2, 2024-09-15T04:00:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:00:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:00:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:00:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:00:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:00:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:00:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:00:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:00:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:00:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:00:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:00:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:00:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:00:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:00:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:00:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:00:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:00:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:00:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:00:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:00:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:00:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:00:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:00:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:00:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:00:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:00:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:00:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:00:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:00:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:00:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:00:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:00:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:51.958 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:51.958 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:51.958 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:00:51.958 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:00:51.958 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:51.958 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:51.958 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:51.958 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:51.958 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:51.958 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:00:51.959 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:00:51.959 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:51.959 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:51.959 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:51.959 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:51.959 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:00:51.959 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:51.959 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:51.959 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:51.959 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:51.959 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:00:51.959 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:51.959 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:51.960 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:51.960 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:51.960 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:00:51.960 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:00:51.960 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:51.960 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:51.960 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:51.960 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:51.960 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:51.960 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:51.960 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:00:51.960 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:00:51.960 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:51.961 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:51.961 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:51.961 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:00:51.961 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:51.961 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:51.961 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:00:51.961 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:51.961 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:51.961 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:51.961 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 3 2024-09-15T04:00:51.961 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:51.962 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:51.962 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:51.962 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:00:51.962 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:00:51.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f54abc34640 1 -- 172.21.15.110:0/3291424069 >> v2:172.21.15.177:6812/28201 conn(0x7f5488038700 msgr2=0x7f548803abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:51.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f54abc34640 1 --2- 172.21.15.110:0/3291424069 >> v2:172.21.15.177:6812/28201 conn(0x7f5488038700 0x7f548803abb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f54a409c5a0 tx=0x7f54a0007e10 comp rx=0 tx=0).stop 2024-09-15T04:00:51.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f54abc34640 10 monclient: shutdown 2024-09-15T04:00:51.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f54abc34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:51.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f54abc34640 1 -- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4136940 msgr2=0x7f54a40975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:51.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f54aa1d1640 1 -- 172.21.15.110:0/3291424069 reap_dead start 2024-09-15T04:00:51.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f54abc34640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4136940 0x7f54a40975a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f549c009f80 tx=0x7f549c036000 comp rx=0 tx=0).stop 2024-09-15T04:00:51.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f54abc34640 1 -- 172.21.15.110:0/3291424069 shutdown_connections 2024-09-15T04:00:51.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f54abc34640 1 --2- 172.21.15.110:0/3291424069 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54a4136940 0x7f54a40975a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:51.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f54abc34640 1 -- 172.21.15.110:0/3291424069 >> 172.21.15.110:0/3291424069 conn(0x7f54a4062f70 msgr2=0x7f54a40a4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:51.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f54abc34640 1 -- 172.21.15.110:0/3291424069 shutdown_connections 2024-09-15T04:00:51.964 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:51.937+0000 7f54abc34640 1 -- 172.21.15.110:0/3291424069 wait complete. 2024-09-15T04:00:51.964 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:00:51.965 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T04:00:51.965 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T04:00:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.057+0000 7f5f9be4f640 10 monclient: get_monmap_and_config 2024-09-15T04:00:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.057+0000 7f5f9be4f640 10 monclient: build_initial_monmap 2024-09-15T04:00:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.057+0000 7f5f9be4f640 10 monclient: monmap: 2024-09-15T04:00:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:00:52.058150+0000 2024-09-15T04:00:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:00:52.058150+0000 2024-09-15T04:00:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:00:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:00:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:00:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:00:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:00:52.073 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.057+0000 7f5f9be4f640 1 Processor -- start 2024-09-15T04:00:52.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.057+0000 7f5f9be4f640 1 -- start start 2024-09-15T04:00:52.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.057+0000 7f5f9be4f640 10 monclient: init 2024-09-15T04:00:52.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f9be4f640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:52.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f9be4f640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-15T04:00:52.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f9be4f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5f940a1a10 0x7f5f94131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f9be4f640 10 monclient(hunting): picked mon.noname-c con 0x7f5f940a1a10 addr v2:172.21.15.177:3300/0 2024-09-15T04:00:52.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f9be4f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5f9409e770 0x7f5f9409ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f9be4f640 10 monclient(hunting): picked mon.noname-a con 0x7f5f9409e770 addr v2:172.21.15.110:3300/0 2024-09-15T04:00:52.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f9be4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f9409f800 addr v1:172.21.15.177:6789/0 2024-09-15T04:00:52.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f9be4f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f940a0850 con 0x7f5f9409f800 2024-09-15T04:00:52.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f9be4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f9be4f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f9409c9a0 con 0x7f5f9409e770 2024-09-15T04:00:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f9be4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f9be4f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f9409cc80 con 0x7f5f940a1a10 2024-09-15T04:00:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f9be4f640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f9be4f640 10 monclient(hunting): authenticate will time out at 875.702535s 2024-09-15T04:00:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5f9409e770 0x7f5f9409ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5f9409e770 0x7f5f9409ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53960/0 (socket says 172.21.15.110:53960) 2024-09-15T04:00:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 1 -- 172.21.15.110:0/1744658014 learned_addr learned my addr 172.21.15.110:0/1744658014 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 10 monclient(hunting): get_auth_request con 0x7f5f9409e770 auth_method 0 2024-09-15T04:00:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:52.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f99beb640 1 --2- 172.21.15.110:0/1744658014 >> v2:172.21.15.177:3300/0 conn(0x7f5f940a1a10 0x7f5f94131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f99beb640 10 monclient(hunting): get_auth_request con 0x7f5f940a1a10 auth_method 0 2024-09-15T04:00:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f99beb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f99beb640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f99beb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 10 monclient(hunting): handle_auth_done global_id 54129 payload 274 2024-09-15T04:00:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 1 -- 172.21.15.110:0/1744658014 >> v2:172.21.15.177:3300/0 conn(0x7f5f940a1a10 msgr2=0x7f5f94131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 1 --2- 172.21.15.110:0/1744658014 >> v2:172.21.15.177:3300/0 conn(0x7f5f940a1a10 0x7f5f94131b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 1 -- 172.21.15.110:0/1744658014 >> v1:172.21.15.177:6789/0 conn(0x7f5f9409f800 legacy=0x7f5f9409fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 10 monclient: _finish_hunting 0 2024-09-15T04:00:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 1 monclient: found mon.noname-a 2024-09-15T04:00:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:00:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.058+0000 7f5f993ea640 1 -- 172.21.15.110:0/1744658014 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9409c3f0 con 0x7f5f9409e770 2024-09-15T04:00:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f993ea640 1 -- 172.21.15.110:0/1744658014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f9409e770 msgr2=0x7f5f9409ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f993ea640 1 --2- 172.21.15.110:0/1744658014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f9409e770 0x7f5f9409ebb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5f940a4d70 tx=0x7f5f880091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e7994e8c68359c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f98be9640 1 -- 172.21.15.110:0/1744658014 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f88032440 con 0x7f5f9409e770 2024-09-15T04:00:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f98be9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f98be9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:00:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f98be9640 10 monclient: dump: 2024-09-15T04:00:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.078 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f98be9640 10 monclient: _finish_auth 0 2024-09-15T04:00:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f98be9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f98be9640 1 -- 172.21.15.110:0/1744658014 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f880325a0 con 0x7f5f9409e770 2024-09-15T04:00:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f9be4f640 5 monclient: authenticate success, global_id 54129 2024-09-15T04:00:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f9be4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f98be9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f9be4f640 10 monclient: get_monmap_and_config success 2024-09-15T04:00:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f9be4f640 4 set_mon_vals no callback set 2024-09-15T04:00:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f98be9640 1 -- 172.21.15.110:0/1744658014 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f88032940 con 0x7f5f9409e770 2024-09-15T04:00:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f98be9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f98be9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.061+0000 7f5f98be9640 10 monclient: dump: 2024-09-15T04:00:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.079 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.062+0000 7f5f9be4f640 10 monclient: shutdown 2024-09-15T04:00:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.062+0000 7f5f9be4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.062+0000 7f5f9be4f640 1 -- 172.21.15.110:0/1744658014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f9409e770 msgr2=0x7f5f9409ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.062+0000 7f5f9be4f640 1 --2- 172.21.15.110:0/1744658014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f9409e770 0x7f5f9409ebb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5f940a4d70 tx=0x7f5f880091d0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.062+0000 7f5f9be4f640 1 -- 172.21.15.110:0/1744658014 shutdown_connections 2024-09-15T04:00:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.062+0000 7f5f9be4f640 1 --2- 172.21.15.110:0/1744658014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f9409e770 0x7f5f9409ebb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.062+0000 7f5f9be4f640 1 --2- 172.21.15.110:0/1744658014 >> v2:172.21.15.177:3300/0 conn(0x7f5f940a1a10 0x7f5f94131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.062+0000 7f5f9be4f640 1 -- 172.21.15.110:0/1744658014 >> 172.21.15.110:0/1744658014 conn(0x7f5f9406e860 msgr2=0x7f5f9406ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.062+0000 7f5f9be4f640 1 -- 172.21.15.110:0/1744658014 shutdown_connections 2024-09-15T04:00:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.062+0000 7f5f9be4f640 1 -- 172.21.15.110:0/1744658014 wait complete. 2024-09-15T04:00:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 10 monclient: build_initial_monmap 2024-09-15T04:00:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 10 monclient: monmap: 2024-09-15T04:00:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:00:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:00:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:00:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:00:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:00:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:00:52.081 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 1 Processor -- start 2024-09-15T04:00:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 1 -- start start 2024-09-15T04:00:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 10 monclient: init 2024-09-15T04:00:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T04:00:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f9409e770 0x7f5f94140d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 10 monclient(hunting): picked mon.noname-a con 0x7f5f9409e770 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f9409f800 0x7f5f94130970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 10 monclient(hunting): picked mon.noname-b con 0x7f5f9409f800 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f940a1a10 0x7f5f9411b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 10 monclient(hunting): picked mon.noname-c con 0x7f5f940a1a10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f94141c10 con 0x7f5f9409e770 2024-09-15T04:00:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f940a11c0 con 0x7f5f9409f800 2024-09-15T04:00:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9411ed80 con 0x7f5f940a1a10 2024-09-15T04:00:52.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:52.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f9be4f640 10 monclient(hunting): authenticate will time out at 875.707906s 2024-09-15T04:00:52.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.063+0000 7f5f99beb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f9409e770 0x7f5f94140d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f9409e770 0x7f5f94140d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53962/0 (socket says 172.21.15.110:53962) 2024-09-15T04:00:52.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 1 -- 172.21.15.110:0/43184355 learned_addr learned my addr 172.21.15.110:0/43184355 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:52.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 10 monclient(hunting): get_auth_request con 0x7f5f9409e770 auth_method 0 2024-09-15T04:00:52.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f9a3ec640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f940a1a10 0x7f5f9411b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f9a3ec640 10 monclient(hunting): get_auth_request con 0x7f5f940a1a10 auth_method 0 2024-09-15T04:00:52.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f9a3ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f9a3ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f9a3ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:52.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:52.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:52.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f9a3ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:52.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f9a3ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:52.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f9a3ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:52.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 10 monclient(hunting): handle_auth_done global_id 54132 payload 824 2024-09-15T04:00:52.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f993ea640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f9409f800 0x7f5f94130970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 1 -- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f940a1a10 msgr2=0x7f5f9411b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f940a1a10 0x7f5f9411b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 1 -- 172.21.15.110:0/43184355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f9409f800 msgr2=0x7f5f94130970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5f9409f800 0x7f5f94130970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 10 monclient: _finish_hunting 0 2024-09-15T04:00:52.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 1 monclient: found mon.noname-a 2024-09-15T04:00:52.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:52.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:00:52.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f99beb640 1 -- 172.21.15.110:0/43184355 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f88009450 con 0x7f5f9409e770 2024-09-15T04:00:52.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.064+0000 7f5f9a3ec640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f940a1a10 0x7f5f9411b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:00:52.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f99beb640 1 -- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f9409e770 msgr2=0x7f5f94140d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:52.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f99beb640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f9409e770 0x7f5f94140d10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c00e1c0 tx=0x7f5f8c009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:52.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f91ffb640 1 -- 172.21.15.110:0/43184355 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c013590 con 0x7f5f9409e770 2024-09-15T04:00:52.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f91ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:00:52.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f91ffb640 10 monclient: dump: 2024-09-15T04:00:52.086 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.086 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.086 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.086 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.086 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.086 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.087 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.087 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.087 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.087 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f91ffb640 10 monclient: _finish_auth 0 2024-09-15T04:00:52.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:52.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f91ffb640 1 -- 172.21.15.110:0/43184355 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8c0136f0 con 0x7f5f9409e770 2024-09-15T04:00:52.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f9be4f640 5 monclient: authenticate success, global_id 54132 2024-09-15T04:00:52.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f91ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:52.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f9be4f640 10 monclient: _renew_subs 2024-09-15T04:00:52.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f91ffb640 1 -- 172.21.15.110:0/43184355 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c013a90 con 0x7f5f9409e770 2024-09-15T04:00:52.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f9be4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:52.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f9be4f640 1 -- 172.21.15.110:0/43184355 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9411cf70 con 0x7f5f9409e770 2024-09-15T04:00:52.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f91ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:52.090 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f91ffb640 10 monclient: dump: 2024-09-15T04:00:52.090 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.090 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.090 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.090 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.090 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.090 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.090 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.091 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f93fff640 4 set_mon_vals no callback set 2024-09-15T04:00:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f9be4f640 10 monclient: _renew_subs 2024-09-15T04:00:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f9be4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.066+0000 7f5f9be4f640 1 -- 172.21.15.110:0/43184355 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9411db40 con 0x7f5f9409e770 2024-09-15T04:00:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f91ffb640 1 -- 172.21.15.110:0/43184355 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f5f8c01e460 con 0x7f5f9409e770 2024-09-15T04:00:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f9be4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f9be4f640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94141250 0x7f5f9411b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.091 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f9be4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f9be4f640 1 -- 172.21.15.110:0/43184355 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9411d540 con 0x7f5f94141250 2024-09-15T04:00:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f91ffb640 1 --2- 172.21.15.110:0/43184355 >> v2:172.21.15.177:6812/28201 conn(0x7f5f80038700 0x7f5f8003abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f91ffb640 1 -- 172.21.15.110:0/43184355 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f5f8c04bd30 con 0x7f5f9409e770 2024-09-15T04:00:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f993ea640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94141250 0x7f5f9411b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f993ea640 10 monclient: get_auth_request con 0x7f5f94141250 auth_method 0 2024-09-15T04:00:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f993ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f993ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f993ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f993ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f993ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.067+0000 7f5f993ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.068+0000 7f5f993ea640 10 monclient(hunting): handle_auth_done global_id 4820 payload 824 2024-09-15T04:00:52.092 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.068+0000 7f5f9a3ec640 1 --2- 172.21.15.110:0/43184355 >> v2:172.21.15.177:6812/28201 conn(0x7f5f80038700 0x7f5f8003abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.068+0000 7f5f993ea640 1 -- 172.21.15.110:0/43184355 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5f9411ef40 con 0x7f5f94141250 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.068+0000 7f5f9a3ec640 10 monclient: get_auth_request con 0x7f5f80038700 auth_method 0 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.070+0000 7f5f993ea640 1 -- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94141250 msgr2=0x7f5f9411b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.070+0000 7f5f993ea640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94141250 0x7f5f9411b9b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5f88034000 tx=0x7f5f88032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.070+0000 7f5f91ffb640 1 -- 172.21.15.110:0/43184355 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f880319f0 con 0x7f5f94141250 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.070+0000 7f5f91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.070+0000 7f5f91ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.070+0000 7f5f91ffb640 10 monclient: dump: 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.093 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.094 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.094 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.094 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.070+0000 7f5f91ffb640 1 -- 172.21.15.110:0/43184355 <== mon.2 v2:172.21.15.110:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f5f88031b50 con 0x7f5f94141250 2024-09-15T04:00:52.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.070+0000 7f5f91ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:52.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.070+0000 7f5f91ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:00:52.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.070+0000 7f5f91ffb640 1 -- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94141250 msgr2=0x7f5f9411b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.070+0000 7f5f91ffb640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94141250 0x7f5f9411b9b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5f88034000 tx=0x7f5f88032000 comp rx=0 tx=0).stop 2024-09-15T04:00:52.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.070+0000 7f5f91ffb640 1 -- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94141250 msgr2=0x7f5f9411b9b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:00:52.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.070+0000 7f5f91ffb640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94141250 0x7f5f9411b9b0 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5f88034000 tx=0x7f5f88032000 comp rx=0 tx=0).stop 2024-09-15T04:00:52.094 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.071+0000 7f5f9a3ec640 1 --2- 172.21.15.110:0/43184355 >> v2:172.21.15.177:6812/28201 conn(0x7f5f80038700 0x7f5f8003abb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5f84007fd0 tx=0x7f5f84007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:52.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.076+0000 7f5f9be4f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:00:52.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.076+0000 7f5f9be4f640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94130eb0 0x7f5f9411bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.096 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.076+0000 7f5f9be4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.076+0000 7f5f9be4f640 1 -- 172.21.15.110:0/43184355 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f94063e80 con 0x7f5f94130eb0 2024-09-15T04:00:52.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.076+0000 7f5f993ea640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94130eb0 0x7f5f9411bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.076+0000 7f5f993ea640 10 monclient: get_auth_request con 0x7f5f94130eb0 auth_method 0 2024-09-15T04:00:52.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.076+0000 7f5f993ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.076+0000 7f5f993ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.076+0000 7f5f993ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.076+0000 7f5f993ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:52.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.076+0000 7f5f993ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:52.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.076+0000 7f5f993ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:52.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.076+0000 7f5f993ea640 10 monclient(hunting): handle_auth_done global_id 4823 payload 824 2024-09-15T04:00:52.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.076+0000 7f5f993ea640 1 -- 172.21.15.110:0/43184355 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5f940647c0 con 0x7f5f94130eb0 2024-09-15T04:00:52.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.078+0000 7f5f993ea640 1 -- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94130eb0 msgr2=0x7f5f9411bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:52.097 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.078+0000 7f5f993ea640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94130eb0 0x7f5f9411bef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5f94063bd0 tx=0x7f5f8803b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:52.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.078+0000 7f5f91ffb640 1 -- 172.21.15.110:0/43184355 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8803b6e0 con 0x7f5f94130eb0 2024-09-15T04:00:52.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.078+0000 7f5f91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.078+0000 7f5f91ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:52.098 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.078+0000 7f5f91ffb640 10 monclient: dump: 2024-09-15T04:00:52.098 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.098 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.098 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.098 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.098 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.098 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.098 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.098 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.098 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.098 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f91ffb640 1 -- 172.21.15.110:0/43184355 <== mon.2 v2:172.21.15.110:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f5f8803b840 con 0x7f5f94130eb0 2024-09-15T04:00:52.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f91ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:00:52.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f91ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:00:52.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f91ffb640 1 -- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94130eb0 msgr2=0x7f5f9411bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f91ffb640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94130eb0 0x7f5f9411bef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5f94063bd0 tx=0x7f5f8803b000 comp rx=0 tx=0).stop 2024-09-15T04:00:52.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f91ffb640 1 -- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94130eb0 msgr2=0x7f5f9411bef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:00:52.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f91ffb640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5f94130eb0 0x7f5f9411bef0 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5f94063bd0 tx=0x7f5f8803b000 comp rx=0 tx=0).stop 2024-09-15T04:00:52.099 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:00:52.099 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:00:52.099 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:00:52.099 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T04:00:52.100 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 22, 2024-09-15T04:00:52.100 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:00:52.100 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:00:52.100 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:00:52.100 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:00:52.100 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:52.100 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 2, 2024-09-15T04:00:52.100 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:00:52.100 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:00:52.100 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:00:52.100 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:52.100 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:52.101 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:00:52.102 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:00:52.103 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:00:52.103 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:00:52.103 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:00:52.103 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:00:52.103 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:00:52.103 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:00:52.103 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:00:52.103 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:00:52.103 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:52.103 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:52.103 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:52.103 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:52.103 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:52.103 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:52.104 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:00:52.105 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:52.105 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.105 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.105 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:52.105 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:00:52.105 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:52.105 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.105 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:52.105 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.105 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:00:52.105 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:00:52.105 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:52.105 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:52.105 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:00:52.106 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:52.107 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.107 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:52.107 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.107 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:00:52.107 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:00:52.107 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:52.107 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:52.107 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:52.107 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.107 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.107 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:00:52.107 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:00:52.107 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:52.107 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:52.108 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.108 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:52.108 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:00:52.108 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:52.108 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.108 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.108 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:52.108 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:00:52.108 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:52.108 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.108 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:52.109 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:00:52.110 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.110 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:52.110 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T04:00:52.110 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.110 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:52.110 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:52.110 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:00:52.110 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.110 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:52.110 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:00:52.110 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.110 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:52.111 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:52.111 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 3 2024-09-15T04:00:52.111 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.111 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:52.111 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.111 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:00:52.111 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:00:52.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f9be4f640 1 -- 172.21.15.110:0/43184355 >> v2:172.21.15.177:6812/28201 conn(0x7f5f80038700 msgr2=0x7f5f8003abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f9be4f640 1 --2- 172.21.15.110:0/43184355 >> v2:172.21.15.177:6812/28201 conn(0x7f5f80038700 0x7f5f8003abb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5f84007fd0 tx=0x7f5f84007e10 comp rx=0 tx=0).stop 2024-09-15T04:00:52.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f9be4f640 10 monclient: shutdown 2024-09-15T04:00:52.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f9be4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:52.112 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f9be4f640 1 -- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f9409e770 msgr2=0x7f5f94140d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f9be4f640 1 --2- 172.21.15.110:0/43184355 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5f9409e770 0x7f5f94140d10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c00e1c0 tx=0x7f5f8c009920 comp rx=0 tx=0).stop 2024-09-15T04:00:52.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f9a3ec640 1 -- 172.21.15.110:0/43184355 reap_dead start 2024-09-15T04:00:52.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f9a3ec640 1 -- 172.21.15.110:0/43184355 reap_dead start 2024-09-15T04:00:52.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f9be4f640 1 -- 172.21.15.110:0/43184355 shutdown_connections 2024-09-15T04:00:52.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f9be4f640 1 -- 172.21.15.110:0/43184355 >> 172.21.15.110:0/43184355 conn(0x7f5f9406e860 msgr2=0x7f5f940a4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:52.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f9be4f640 1 -- 172.21.15.110:0/43184355 shutdown_connections 2024-09-15T04:00:52.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.079+0000 7f5f9be4f640 1 -- 172.21.15.110:0/43184355 wait complete. 2024-09-15T04:00:52.113 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:00:52.113 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T04:00:52.114 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T04:00:52.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.208+0000 7f641bcd4640 10 monclient: get_monmap_and_config 2024-09-15T04:00:52.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.208+0000 7f641bcd4640 10 monclient: build_initial_monmap 2024-09-15T04:00:52.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.208+0000 7f641bcd4640 10 monclient: monmap: 2024-09-15T04:00:52.209 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:52.209 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.209 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:00:52.209014+0000 2024-09-15T04:00:52.210 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:00:52.209014+0000 2024-09-15T04:00:52.210 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:52.210 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.210 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:00:52.210 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:00:52.210 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:00:52.210 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:00:52.210 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:00:52.210 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.208+0000 7f641bcd4640 1 Processor -- start 2024-09-15T04:00:52.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.208+0000 7f641bcd4640 1 -- start start 2024-09-15T04:00:52.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.208+0000 7f641bcd4640 10 monclient: init 2024-09-15T04:00:52.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641bcd4640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:52.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641bcd4640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-15T04:00:52.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641bcd4640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f64140a1cf0 0x7f641409e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641bcd4640 10 monclient(hunting): picked mon.noname-c con 0x7f64140a1cf0 addr v2:172.21.15.177:3300/0 2024-09-15T04:00:52.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641bcd4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6414131a80 0x7f641409eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641bcd4640 10 monclient(hunting): picked mon.noname-a con 0x7f6414131a80 addr v2:172.21.15.110:3300/0 2024-09-15T04:00:52.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641bcd4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f641409f760 addr v1:172.21.15.110:6789/0 2024-09-15T04:00:52.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641bcd4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f641409cb70 con 0x7f641409f760 2024-09-15T04:00:52.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641bcd4640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641bcd4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64140a20c0 con 0x7f6414131a80 2024-09-15T04:00:52.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641bcd4640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641bcd4640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f641409c330 con 0x7f64140a1cf0 2024-09-15T04:00:52.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641bcd4640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:52.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641bcd4640 10 monclient(hunting): authenticate will time out at 875.853477s 2024-09-15T04:00:52.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641926f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6414131a80 0x7f641409eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641926f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6414131a80 0x7f641409eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53970/0 (socket says 172.21.15.110:53970) 2024-09-15T04:00:52.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641926f640 1 -- 172.21.15.110:0/630653206 learned_addr learned my addr 172.21.15.110:0/630653206 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:52.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641926f640 10 monclient(hunting): get_auth_request con 0x7f6414131a80 auth_method 0 2024-09-15T04:00:52.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641926f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641926f640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.222 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641926f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641926f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:52.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641926f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:52.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 1 --2- 172.21.15.110:0/630653206 >> v2:172.21.15.177:3300/0 conn(0x7f64140a1cf0 0x7f641409e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f641926f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:52.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 10 monclient(hunting): get_auth_request con 0x7f64140a1cf0 auth_method 0 2024-09-15T04:00:52.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.223 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:52.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:52.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:52.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 10 monclient(hunting): handle_auth_done global_id 4801 payload 274 2024-09-15T04:00:52.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 1 -- 172.21.15.110:0/630653206 >> v2:172.21.15.110:3300/0 conn(0x7f6414131a80 msgr2=0x7f641409eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:52.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 1 --2- 172.21.15.110:0/630653206 >> v2:172.21.15.110:3300/0 conn(0x7f6414131a80 0x7f641409eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 1 -- 172.21.15.110:0/630653206 >> v1:172.21.15.110:6789/0 conn(0x7f641409f760 legacy=0x7f641409fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:52.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 10 monclient: _finish_hunting 0 2024-09-15T04:00:52.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 1 monclient: found mon.noname-c 2024-09-15T04:00:52.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:52.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T04:00:52.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.209+0000 7f6419a70640 1 -- 172.21.15.110:0/630653206 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6414131fb0 con 0x7f64140a1cf0 2024-09-15T04:00:52.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.212+0000 7f6419a70640 1 -- 172.21.15.110:0/630653206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f64140a1cf0 msgr2=0x7f641409e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:52.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.212+0000 7f6419a70640 1 --2- 172.21.15.110:0/630653206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f64140a1cf0 0x7f641409e660 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6414063d70 tx=0x7f64100090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44c2780be3a5e2f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:52.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f6418a6e640 1 -- 172.21.15.110:0/630653206 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641002ae00 con 0x7f64140a1cf0 2024-09-15T04:00:52.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f6418a6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f6418a6e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:00:52.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f6418a6e640 10 monclient: dump: 2024-09-15T04:00:52.225 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.225 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.225 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.225 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.225 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.225 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.226 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.226 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.226 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.226 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f6418a6e640 10 monclient: _finish_auth 0 2024-09-15T04:00:52.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f6418a6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:52.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f6418a6e640 1 -- 172.21.15.110:0/630653206 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6410032440 con 0x7f64140a1cf0 2024-09-15T04:00:52.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f641bcd4640 5 monclient: authenticate success, global_id 4801 2024-09-15T04:00:52.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f641bcd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:52.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f6418a6e640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:52.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f641bcd4640 10 monclient: get_monmap_and_config success 2024-09-15T04:00:52.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f6418a6e640 1 -- 172.21.15.110:0/630653206 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64100327e0 con 0x7f64140a1cf0 2024-09-15T04:00:52.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f641bcd4640 4 set_mon_vals no callback set 2024-09-15T04:00:52.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f6418a6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f6418a6e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:00:52.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f6418a6e640 10 monclient: dump: 2024-09-15T04:00:52.227 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.227 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.227 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.227 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.227 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.227 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.228 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.228 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.228 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.228 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f641bcd4640 10 monclient: shutdown 2024-09-15T04:00:52.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f641bcd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:52.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f641bcd4640 1 -- 172.21.15.110:0/630653206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f64140a1cf0 msgr2=0x7f641409e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f641bcd4640 1 --2- 172.21.15.110:0/630653206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f64140a1cf0 0x7f641409e660 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6414063d70 tx=0x7f64100090f0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f641bcd4640 1 -- 172.21.15.110:0/630653206 shutdown_connections 2024-09-15T04:00:52.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f641bcd4640 1 --2- 172.21.15.110:0/630653206 >> v2:172.21.15.110:3300/0 conn(0x7f6414131a80 0x7f641409eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f641bcd4640 1 --2- 172.21.15.110:0/630653206 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f64140a1cf0 0x7f641409e660 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f641bcd4640 1 -- 172.21.15.110:0/630653206 >> 172.21.15.110:0/630653206 conn(0x7f6414062f70 msgr2=0x7f6414063340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:52.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f641bcd4640 1 -- 172.21.15.110:0/630653206 shutdown_connections 2024-09-15T04:00:52.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.213+0000 7f641bcd4640 1 -- 172.21.15.110:0/630653206 wait complete. 2024-09-15T04:00:52.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 10 monclient: build_initial_monmap 2024-09-15T04:00:52.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 10 monclient: monmap: 2024-09-15T04:00:52.230 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:52.230 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:00:52.230 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:00:52.230 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:00:52.230 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:52.230 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.230 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:00:52.230 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:00:52.230 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:00:52.230 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 1 Processor -- start 2024-09-15T04:00:52.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 1 -- start start 2024-09-15T04:00:52.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 10 monclient: init 2024-09-15T04:00:52.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:52.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T04:00:52.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641409f760 0x7f641409a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 10 monclient(hunting): picked mon.noname-b con 0x7f641409f760 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:52.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64140a1cf0 0x7f6414130750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 10 monclient(hunting): picked mon.noname-a con 0x7f64140a1cf0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:52.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6414131a80 0x7f641411b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 10 monclient(hunting): picked mon.noname-c con 0x7f6414131a80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:52.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641409b470 con 0x7f64140a1cf0 2024-09-15T04:00:52.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641411d870 con 0x7f641409f760 2024-09-15T04:00:52.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641411db50 con 0x7f6414131a80 2024-09-15T04:00:52.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:52.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641bcd4640 10 monclient(hunting): authenticate will time out at 875.858920s 2024-09-15T04:00:52.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.214+0000 7f641926f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64140a1cf0 0x7f6414130750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64140a1cf0 0x7f6414130750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:53984/0 (socket says 172.21.15.110:53984) 2024-09-15T04:00:52.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 1 -- 172.21.15.110:0/766452893 learned_addr learned my addr 172.21.15.110:0/766452893 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:52.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 10 monclient(hunting): get_auth_request con 0x7f64140a1cf0 auth_method 0 2024-09-15T04:00:52.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641a271640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6414131a80 0x7f641411b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641a271640 10 monclient(hunting): get_auth_request con 0x7f6414131a80 auth_method 0 2024-09-15T04:00:52.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641a271640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641a271640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641a271640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:52.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:52.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:52.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641a271640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:52.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641a271640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:52.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641a271640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:52.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f6419a70640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641409f760 0x7f641409a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f6419a70640 10 monclient(hunting): get_auth_request con 0x7f641409f760 auth_method 0 2024-09-15T04:00:52.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f6419a70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f6419a70640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f6419a70640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 10 monclient(hunting): handle_auth_done global_id 54138 payload 824 2024-09-15T04:00:52.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 1 -- 172.21.15.110:0/766452893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6414131a80 msgr2=0x7f641411b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6414131a80 0x7f641411b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 1 -- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641409f760 msgr2=0x7f641409a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641409f760 0x7f641409a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 10 monclient: _finish_hunting 0 2024-09-15T04:00:52.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 1 monclient: found mon.noname-a 2024-09-15T04:00:52.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:52.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:00:52.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641926f640 1 -- 172.21.15.110:0/766452893 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6410009390 con 0x7f64140a1cf0 2024-09-15T04:00:52.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f6419a70640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641409f760 0x7f641409a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T04:00:52.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.215+0000 7f641a271640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6414131a80 0x7f641411b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:00:52.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f641926f640 1 -- 172.21.15.110:0/766452893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64140a1cf0 msgr2=0x7f6414130750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:52.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f641926f640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64140a1cf0 0x7f6414130750 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f640400ea20 tx=0x7f64040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:52.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f6409ffb640 1 -- 172.21.15.110:0/766452893 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64040095a0 con 0x7f64140a1cf0 2024-09-15T04:00:52.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f6409ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f6409ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:00:52.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f6409ffb640 10 monclient: dump: 2024-09-15T04:00:52.235 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.235 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.235 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.235 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.236 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.236 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.236 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.236 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.236 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.236 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f6409ffb640 10 monclient: _finish_auth 0 2024-09-15T04:00:52.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f6409ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:52.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f6409ffb640 1 -- 172.21.15.110:0/766452893 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640400cb50 con 0x7f64140a1cf0 2024-09-15T04:00:52.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f641bcd4640 5 monclient: authenticate success, global_id 54138 2024-09-15T04:00:52.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f641bcd4640 10 monclient: _renew_subs 2024-09-15T04:00:52.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f641bcd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:52.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f641bcd4640 1 -- 172.21.15.110:0/766452893 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f641411de30 con 0x7f64140a1cf0 2024-09-15T04:00:52.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f6409ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:52.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f6409ffb640 1 -- 172.21.15.110:0/766452893 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404013470 con 0x7f64140a1cf0 2024-09-15T04:00:52.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f6409ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f6409ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:52.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f6409ffb640 10 monclient: dump: 2024-09-15T04:00:52.237 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.237 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.237 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.238 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.238 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.238 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.238 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.238 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.238 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.238 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f641bcd4640 10 monclient: _renew_subs 2024-09-15T04:00:52.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f640bfff640 4 set_mon_vals no callback set 2024-09-15T04:00:52.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f641bcd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:52.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.217+0000 7f641bcd4640 1 -- 172.21.15.110:0/766452893 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6414141a80 con 0x7f64140a1cf0 2024-09-15T04:00:52.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.218+0000 7f641bcd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:52.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.218+0000 7f641bcd4640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641411c910 0x7f641411b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.218+0000 7f641bcd4640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.218+0000 7f641bcd4640 1 -- 172.21.15.110:0/766452893 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f641406f1e0 con 0x7f641411c910 2024-09-15T04:00:52.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.218+0000 7f6409ffb640 1 -- 172.21.15.110:0/766452893 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f6404013820 con 0x7f64140a1cf0 2024-09-15T04:00:52.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.218+0000 7f6409ffb640 1 --2- 172.21.15.110:0/766452893 >> v2:172.21.15.177:6812/28201 conn(0x7f63f8038700 0x7f63f803abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.219+0000 7f6409ffb640 1 -- 172.21.15.110:0/766452893 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f640404b870 con 0x7f64140a1cf0 2024-09-15T04:00:52.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.219+0000 7f6419a70640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641411c910 0x7f641411b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.219+0000 7f6419a70640 10 monclient: get_auth_request con 0x7f641411c910 auth_method 0 2024-09-15T04:00:52.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.219+0000 7f6419a70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.219+0000 7f6419a70640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.219+0000 7f6419a70640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.219+0000 7f6419a70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:52.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.219+0000 7f6419a70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:52.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.219+0000 7f641a271640 1 --2- 172.21.15.110:0/766452893 >> v2:172.21.15.177:6812/28201 conn(0x7f63f8038700 0x7f63f803abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.219+0000 7f6419a70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:52.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.219+0000 7f641a271640 10 monclient: get_auth_request con 0x7f63f8038700 auth_method 0 2024-09-15T04:00:52.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.219+0000 7f6419a70640 10 monclient(hunting): handle_auth_done global_id 4807 payload 824 2024-09-15T04:00:52.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.219+0000 7f6419a70640 1 -- 172.21.15.110:0/766452893 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f64140647e0 con 0x7f641411c910 2024-09-15T04:00:52.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.221+0000 7f6419a70640 1 -- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641411c910 msgr2=0x7f641411b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:52.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.221+0000 7f6419a70640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641411c910 0x7f641411b910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6410034000 tx=0x7f6410032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:52.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.221+0000 7f641a271640 1 --2- 172.21.15.110:0/766452893 >> v2:172.21.15.177:6812/28201 conn(0x7f63f8038700 0x7f63f803abb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f641411e150 tx=0x7f640c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:52.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.221+0000 7f6409ffb640 1 -- 172.21.15.110:0/766452893 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6410031d30 con 0x7f641411c910 2024-09-15T04:00:52.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.221+0000 7f6409ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.221+0000 7f6409ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:00:52.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.221+0000 7f6409ffb640 10 monclient: dump: 2024-09-15T04:00:52.244 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.244 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.244 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.244 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.244 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.244 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.244 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.245 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.245 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.245 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.222+0000 7f6409ffb640 1 -- 172.21.15.110:0/766452893 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6410031e90 con 0x7f641411c910 2024-09-15T04:00:52.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.222+0000 7f6409ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:52.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.222+0000 7f6409ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:00:52.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.222+0000 7f6409ffb640 1 -- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641411c910 msgr2=0x7f641411b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.222+0000 7f6409ffb640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641411c910 0x7f641411b910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6410034000 tx=0x7f6410032000 comp rx=0 tx=0).stop 2024-09-15T04:00:52.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.222+0000 7f6409ffb640 1 -- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641411c910 msgr2=0x7f641411b910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:00:52.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.222+0000 7f6409ffb640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f641411c910 0x7f641411b910 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.227+0000 7f641bcd4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:00:52.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.227+0000 7f641bcd4640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6414130c90 0x7f641411be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.227+0000 7f641bcd4640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.227+0000 7f641bcd4640 1 -- 172.21.15.110:0/766452893 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6414067130 con 0x7f6414130c90 2024-09-15T04:00:52.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.227+0000 7f6419a70640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6414130c90 0x7f641411be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.228+0000 7f6419a70640 10 monclient: get_auth_request con 0x7f6414130c90 auth_method 0 2024-09-15T04:00:52.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.228+0000 7f6419a70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.228+0000 7f6419a70640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.228+0000 7f6419a70640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.228+0000 7f6419a70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:52.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.228+0000 7f6419a70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:52.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.228+0000 7f6419a70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:52.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.228+0000 7f6419a70640 10 monclient(hunting): handle_auth_done global_id 4810 payload 824 2024-09-15T04:00:52.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.228+0000 7f6419a70640 1 -- 172.21.15.110:0/766452893 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f641414a070 con 0x7f6414130c90 2024-09-15T04:00:52.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.230+0000 7f6419a70640 1 -- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6414130c90 msgr2=0x7f641411be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:52.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.230+0000 7f6419a70640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6414130c90 0x7f641411be50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f641406ef70 tx=0x7f641003c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:52.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.230+0000 7f6409ffb640 1 -- 172.21.15.110:0/766452893 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6410047430 con 0x7f6414130c90 2024-09-15T04:00:52.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.230+0000 7f6409ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.230+0000 7f6409ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:00:52.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.230+0000 7f6409ffb640 10 monclient: dump: 2024-09-15T04:00:52.247 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.247 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.248 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.248 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.248 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.248 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.248 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.248 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.248 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.248 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.230+0000 7f6409ffb640 1 -- 172.21.15.110:0/766452893 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f641004b070 con 0x7f6414130c90 2024-09-15T04:00:52.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.230+0000 7f6409ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:00:52.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.230+0000 7f6409ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:00:52.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.230+0000 7f6409ffb640 1 -- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6414130c90 msgr2=0x7f641411be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.230+0000 7f6409ffb640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6414130c90 0x7f641411be50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f641406ef70 tx=0x7f641003c000 comp rx=0 tx=0).stop 2024-09-15T04:00:52.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.230+0000 7f6409ffb640 1 -- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6414130c90 msgr2=0x7f641411be50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:00:52.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.230+0000 7f6409ffb640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6414130c90 0x7f641411be50 secure :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f641406ef70 tx=0x7f641003c000 comp rx=0 tx=0).stop 2024-09-15T04:00:52.249 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:00:52.249 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:00:52.249 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:00:52.249 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T04:00:52.249 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 22, 2024-09-15T04:00:52.249 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:00:52.249 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:00:52.250 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:00:52.250 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:00:52.250 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:52.250 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 2, 2024-09-15T04:00:52.250 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:00:52.250 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:00:52.250 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:00:52.250 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:52.250 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:52.250 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:52.250 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:52.250 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:52.250 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:52.251 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:52.251 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:52.251 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:52.251 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:52.251 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:00:52.251 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:00:52.251 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:52.251 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:52.251 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:52.251 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:52.251 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:52.251 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:52.251 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:52.251 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:52.252 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:52.252 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:52.252 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.252 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:00:52.252 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:00:52.252 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:00:52.252 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:00:52.252 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:00:52.252 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:00:52.252 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:00:52.252 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:00:52.253 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:00:52.253 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:00:52.253 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:00:52.253 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:00:52.253 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:00:52.253 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:00:52.253 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:00:52.253 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:00:52.253 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:00:52.253 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:52.253 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:52.253 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:52.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:52.254 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:52.254 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:52.254 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:52.254 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:52.254 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:52.254 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:52.254 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:00:52.255 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.255 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:00:52.255 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.255 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:00:52.255 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:00:52.255 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:52.255 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:52.255 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.256 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:52.256 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:00:52.256 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:52.256 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.256 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.256 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:52.256 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:00:52.256 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:52.256 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.257 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:52.257 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.257 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:00:52.257 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:00:52.257 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:52.257 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:52.257 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:52.257 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.257 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.257 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:00:52.258 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:00:52.258 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:52.258 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:52.258 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.258 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:52.258 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:00:52.258 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:52.258 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.258 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.258 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:52.258 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:00:52.259 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:52.259 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.259 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:52.259 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.259 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:00:52.259 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:00:52.259 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:52.259 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:52.259 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:52.259 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.259 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.259 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:00:52.259 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:00:52.260 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:52.260 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:52.260 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.260 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:52.260 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:00:52.260 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:52.260 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.260 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.261 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:52.261 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:00:52.261 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:52.261 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.261 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:52.261 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.261 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:00:52.261 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:00:52.261 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:52.261 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:52.261 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:52.261 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.262 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:52.262 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.262 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:00:52.262 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:00:52.262 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.262 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:52.262 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:52.262 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:00:52.262 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.263 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:52.263 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T04:00:52.263 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.263 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:52.263 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:52.263 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 5 2024-09-15T04:00:52.263 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.263 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:52.263 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:00:52.263 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.263 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:52.263 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:52.263 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 5 2024-09-15T04:00:52.264 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.264 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:52.264 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": [ 2024-09-15T04:00:52.264 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:52.264 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:52.264 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:52.265 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:00:52.265 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:52.265 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:52.265 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:52.265 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:00:52.265 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:00:52.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.231+0000 7f641bcd4640 1 -- 172.21.15.110:0/766452893 >> v2:172.21.15.177:6812/28201 conn(0x7f63f8038700 msgr2=0x7f63f803abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.231+0000 7f641bcd4640 1 --2- 172.21.15.110:0/766452893 >> v2:172.21.15.177:6812/28201 conn(0x7f63f8038700 0x7f63f803abb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f641411e150 tx=0x7f640c008040 comp rx=0 tx=0).stop 2024-09-15T04:00:52.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.231+0000 7f641bcd4640 10 monclient: shutdown 2024-09-15T04:00:52.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.231+0000 7f641bcd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:52.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.231+0000 7f641bcd4640 1 -- 172.21.15.110:0/766452893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64140a1cf0 msgr2=0x7f6414130750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.231+0000 7f641a271640 1 -- 172.21.15.110:0/766452893 reap_dead start 2024-09-15T04:00:52.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.231+0000 7f641bcd4640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64140a1cf0 0x7f6414130750 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f640400ea20 tx=0x7f64040091d0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.231+0000 7f641bcd4640 1 -- 172.21.15.110:0/766452893 shutdown_connections 2024-09-15T04:00:52.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.231+0000 7f641bcd4640 1 --2- 172.21.15.110:0/766452893 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f64140a1cf0 0x7f6414130750 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.231+0000 7f641bcd4640 1 -- 172.21.15.110:0/766452893 >> 172.21.15.110:0/766452893 conn(0x7f6414062f70 msgr2=0x7f64141349e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:52.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.231+0000 7f641bcd4640 1 -- 172.21.15.110:0/766452893 shutdown_connections 2024-09-15T04:00:52.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.231+0000 7f641bcd4640 1 -- 172.21.15.110:0/766452893 wait complete. 2024-09-15T04:00:52.270 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:00:52.270 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T04:00:52.270 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T04:00:52.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.362+0000 7f9068709640 10 monclient: get_monmap_and_config 2024-09-15T04:00:52.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.362+0000 7f9068709640 10 monclient: build_initial_monmap 2024-09-15T04:00:52.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.362+0000 7f9068709640 10 monclient: monmap: 2024-09-15T04:00:52.363 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:52.363 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.363 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:00:52.363060+0000 2024-09-15T04:00:52.363 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:00:52.363060+0000 2024-09-15T04:00:52.363 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:52.363 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.364 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:00:52.364 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:00:52.364 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:00:52.364 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:00:52.364 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:00:52.364 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.362+0000 7f9068709640 1 Processor -- start 2024-09-15T04:00:52.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.362+0000 7f9068709640 1 -- start start 2024-09-15T04:00:52.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.362+0000 7f9068709640 10 monclient: init 2024-09-15T04:00:52.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9068709640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:52.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9068709640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-15T04:00:52.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9068709640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 0x7f906009d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9068709640 10 monclient(hunting): picked mon.noname-b con 0x7f90600a1a60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:52.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9068709640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f906009f870 addr v1:172.21.15.177:6789/0 2024-09-15T04:00:52.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9068709640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f90600a0350 0x7f9060138b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9068709640 10 monclient(hunting): picked mon.noname-c con 0x7f90600a0350 addr v2:172.21.15.177:3300/0 2024-09-15T04:00:52.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9068709640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90600a1260 con 0x7f906009f870 2024-09-15T04:00:52.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9068709640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9068709640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f906009cdf0 con 0x7f90600a0350 2024-09-15T04:00:52.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9068709640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9068709640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906009c870 con 0x7f90600a1a60 2024-09-15T04:00:52.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9068709640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:52.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9068709640 10 monclient(hunting): authenticate will time out at 876.007324s 2024-09-15T04:00:52.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 0x7f906009d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 0x7f906009d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50738/0 (socket says 172.21.15.110:50738) 2024-09-15T04:00:52.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 1 -- 172.21.15.110:0/596275085 learned_addr learned my addr 172.21.15.110:0/596275085 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:52.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 10 monclient(hunting): get_auth_request con 0x7f90600a1a60 auth_method 0 2024-09-15T04:00:52.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:52.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:52.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:52.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f9066ca6640 1 --2- 172.21.15.110:0/596275085 >> v2:172.21.15.177:3300/0 conn(0x7f90600a0350 0x7f9060138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 10 monclient(hunting): handle_auth_done global_id 4829 payload 274 2024-09-15T04:00:52.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 1 -- 172.21.15.110:0/596275085 >> v2:172.21.15.177:3300/0 conn(0x7f90600a0350 msgr2=0x7f9060138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:52.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 1 --2- 172.21.15.110:0/596275085 >> v2:172.21.15.177:3300/0 conn(0x7f90600a0350 0x7f9060138b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 1 -- 172.21.15.110:0/596275085 >> v1:172.21.15.177:6789/0 conn(0x7f906009f870 legacy=0x7f906009fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:52.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 10 monclient: _finish_hunting 0 2024-09-15T04:00:52.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 1 monclient: found mon.noname-b 2024-09-15T04:00:52.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:52.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:00:52.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.363+0000 7f90664a5640 1 -- 172.21.15.110:0/596275085 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90600a0c60 con 0x7f90600a1a60 2024-09-15T04:00:52.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f90664a5640 1 -- 172.21.15.110:0/596275085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 msgr2=0x7f906009d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:52.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f90664a5640 1 --2- 172.21.15.110:0/596275085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 0x7f906009d4f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f905c009910 tx=0x7f905c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96297b03a1f2bf15 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:52.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f90654a3640 1 -- 172.21.15.110:0/596275085 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c032440 con 0x7f90600a1a60 2024-09-15T04:00:52.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f90654a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f90654a3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:00:52.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f90654a3640 10 monclient: dump: 2024-09-15T04:00:52.371 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.371 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.371 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.371 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.372 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.372 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.372 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.372 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.372 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.372 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f90654a3640 10 monclient: _finish_auth 0 2024-09-15T04:00:52.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f90654a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:52.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f90654a3640 1 -- 172.21.15.110:0/596275085 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905c0325a0 con 0x7f90600a1a60 2024-09-15T04:00:52.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f9068709640 5 monclient: authenticate success, global_id 4829 2024-09-15T04:00:52.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f9068709640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:52.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f90654a3640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:52.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f9068709640 10 monclient: get_monmap_and_config success 2024-09-15T04:00:52.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f9068709640 4 set_mon_vals no callback set 2024-09-15T04:00:52.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f90654a3640 1 -- 172.21.15.110:0/596275085 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c032940 con 0x7f90600a1a60 2024-09-15T04:00:52.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f90654a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f90654a3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:52.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.366+0000 7f90654a3640 10 monclient: dump: 2024-09-15T04:00:52.373 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.373 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.373 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.373 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.373 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.373 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.373 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.373 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.367+0000 7f9068709640 10 monclient: shutdown 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.367+0000 7f9068709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.367+0000 7f9068709640 1 -- 172.21.15.110:0/596275085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 msgr2=0x7f906009d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.367+0000 7f9068709640 1 --2- 172.21.15.110:0/596275085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 0x7f906009d4f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f905c009910 tx=0x7f905c0091d0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.367+0000 7f9068709640 1 -- 172.21.15.110:0/596275085 shutdown_connections 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.367+0000 7f9068709640 1 --2- 172.21.15.110:0/596275085 >> v2:172.21.15.177:3300/0 conn(0x7f90600a0350 0x7f9060138b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.367+0000 7f9068709640 1 --2- 172.21.15.110:0/596275085 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 0x7f906009d4f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.367+0000 7f9068709640 1 -- 172.21.15.110:0/596275085 >> 172.21.15.110:0/596275085 conn(0x7f9060063270 msgr2=0x7f9060063670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.367+0000 7f9068709640 1 -- 172.21.15.110:0/596275085 shutdown_connections 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.367+0000 7f9068709640 1 -- 172.21.15.110:0/596275085 wait complete. 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.367+0000 7f9068709640 10 monclient: build_initial_monmap 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.367+0000 7f9068709640 10 monclient: monmap: 2024-09-15T04:00:52.374 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:52.375 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:00:52.375 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:00:52.375 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:00:52.375 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:52.375 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.375 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:00:52.375 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:00:52.375 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:00:52.375 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 1 Processor -- start 2024-09-15T04:00:52.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 1 -- start start 2024-09-15T04:00:52.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 10 monclient: init 2024-09-15T04:00:52.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:52.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T04:00:52.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f906009f870 0x7f9060099b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 10 monclient(hunting): picked mon.noname-a con 0x7f906009f870 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:52.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90600a0350 0x7f906009a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 10 monclient(hunting): picked mon.noname-b con 0x7f90600a0350 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:52.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 0x7f906009a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 10 monclient(hunting): picked mon.noname-c con 0x7f90600a1a60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:52.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906011d680 con 0x7f906009f870 2024-09-15T04:00:52.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906011d3a0 con 0x7f90600a0350 2024-09-15T04:00:52.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:52.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f906011d960 con 0x7f90600a1a60 2024-09-15T04:00:52.380 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9068709640 10 monclient(hunting): authenticate will time out at 876.012535s 2024-09-15T04:00:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 0x7f906009a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f90664a5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f906009f870 0x7f9060099b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 0x7f906009a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50742/0 (socket says 172.21.15.110:50742) 2024-09-15T04:00:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 1 -- 172.21.15.110:0/2495896985 learned_addr learned my addr 172.21.15.110:0/2495896985 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 10 monclient(hunting): get_auth_request con 0x7f90600a1a60 auth_method 0 2024-09-15T04:00:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f90664a5640 10 monclient(hunting): get_auth_request con 0x7f906009f870 auth_method 0 2024-09-15T04:00:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f90664a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f90664a5640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:52.381 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f90664a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f90664a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f90664a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f90664a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9065ca4640 1 --2- 172.21.15.110:0/2495896985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90600a0350 0x7f906009a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 10 monclient(hunting): handle_auth_done global_id 4832 payload 824 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 1 -- 172.21.15.110:0/2495896985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90600a0350 msgr2=0x7f906009a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 1 --2- 172.21.15.110:0/2495896985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90600a0350 0x7f906009a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 1 -- 172.21.15.110:0/2495896985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f906009f870 msgr2=0x7f9060099b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 1 --2- 172.21.15.110:0/2495896985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f906009f870 0x7f9060099b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 10 monclient: _finish_hunting 0 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 1 monclient: found mon.noname-c 2024-09-15T04:00:52.382 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:52.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:00:52.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9066ca6640 1 -- 172.21.15.110:0/2495896985 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c009450 con 0x7f90600a1a60 2024-09-15T04:00:52.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9065ca4640 10 monclient: get_auth_request con 0x7f90600a0350 auth_method 0 2024-09-15T04:00:52.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.368+0000 7f9065ca4640 1 --2- 172.21.15.110:0/2495896985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90600a0350 0x7f906009a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:00:52.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f9066ca6640 1 -- 172.21.15.110:0/2495896985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 msgr2=0x7f906009a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:52.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f9066ca6640 1 --2- 172.21.15.110:0/2495896985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 0x7f906009a580 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f905800ceb0 tx=0x7f9058007870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:52.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f90567fc640 1 -- 172.21.15.110:0/2495896985 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905800e610 con 0x7f90600a1a60 2024-09-15T04:00:52.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f90567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f90567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:00:52.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f90567fc640 10 monclient: dump: 2024-09-15T04:00:52.383 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.383 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.383 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.383 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.384 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f90567fc640 10 monclient: _finish_auth 0 2024-09-15T04:00:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f90567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f90567fc640 1 -- 172.21.15.110:0/2495896985 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905800e770 con 0x7f90600a1a60 2024-09-15T04:00:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f9068709640 5 monclient: authenticate success, global_id 4832 2024-09-15T04:00:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f90567fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:52.384 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f90567fc640 1 -- 172.21.15.110:0/2495896985 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905800eb30 con 0x7f90600a1a60 2024-09-15T04:00:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f90567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f90567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f90567fc640 10 monclient: dump: 2024-09-15T04:00:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:52.385 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:52.386 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f9068709640 10 monclient: _renew_subs 2024-09-15T04:00:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f9068709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f9068709640 1 -- 172.21.15.110:0/2495896985 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90601343e0 con 0x7f90600a1a60 2024-09-15T04:00:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f9064ca2640 4 set_mon_vals no callback set 2024-09-15T04:00:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f9068709640 10 monclient: _renew_subs 2024-09-15T04:00:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f9068709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.371+0000 7f9068709640 1 -- 172.21.15.110:0/2495896985 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f906009ac60 con 0x7f90600a1a60 2024-09-15T04:00:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.372+0000 7f90567fc640 1 -- 172.21.15.110:0/2495896985 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f9058009030 con 0x7f90600a1a60 2024-09-15T04:00:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.372+0000 7f9068709640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.372+0000 7f9068709640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.372+0000 7f9068709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.372+0000 7f90567fc640 1 --2- 172.21.15.110:0/2495896985 >> v2:172.21.15.177:6812/28201 conn(0x7f9044038700 0x7f904403abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:52.386 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.372+0000 7f9068709640 1 -- 172.21.15.110:0/2495896985 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90600a14c0 con 0x7f90600a1a60 2024-09-15T04:00:52.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.372+0000 7f90567fc640 1 -- 172.21.15.110:0/2495896985 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9058012070 con 0x7f90600a1a60 2024-09-15T04:00:52.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.372+0000 7f90664a5640 1 --2- 172.21.15.110:0/2495896985 >> v2:172.21.15.177:6812/28201 conn(0x7f9044038700 0x7f904403abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:52.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.372+0000 7f90664a5640 10 monclient: get_auth_request con 0x7f9044038700 auth_method 0 2024-09-15T04:00:52.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.375+0000 7f90664a5640 1 --2- 172.21.15.110:0/2495896985 >> v2:172.21.15.177:6812/28201 conn(0x7f9044038700 0x7f904403abb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9060068040 tx=0x7f905c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:52.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.376+0000 7f90567fc640 1 -- 172.21.15.110:0/2495896985 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f90600a14c0 con 0x7f90600a1a60 2024-09-15T04:00:52.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.376+0000 7f90567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:52.387 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.376+0000 7f90567fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:00:52.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.590+0000 7f9068709640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T04:00:52.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.590+0000 7f9068709640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T04:00:52.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.590+0000 7f9068709640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:52.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.590+0000 7f9068709640 1 -- 172.21.15.110:0/2495896985 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f90600a14c0 con 0x7f90600a1a60 2024-09-15T04:00:52.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.591+0000 7f90567fc640 1 -- 172.21.15.110:0/2495896985 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f90600a14c0 con 0x7f90600a1a60 2024-09-15T04:00:52.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.591+0000 7f90567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T04:00:52.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.591+0000 7f90567fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:00:52.594 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.594 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=8,logm=67} crc {auth=4197603444,config=3140514009,health=2809202031,kv=1242700903,logm=2166651776}) 2024-09-15T04:00:52.594 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=1,health=12,kv=8,logm=67} crc {auth=4197603444,config=1228700967,health=2056321573,kv=1242700903,logm=2309333674}) 2024-09-15T04:00:52.594 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.594 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=8,logm=67} crc {auth=4197603444,config=3140514009,health=2809202031,kv=1242700903,logm=2166651776}) 2024-09-15T04:00:52.594 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.592+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=11,kv=7,logm=69} crc {auth=1969902965,config=3140514009,health=588522927,kv=1242700903,logm=2190167730}) 2024-09-15T04:00:52.594 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.592+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.592+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3400636892}) 2024-09-15T04:00:52.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.592+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=707501623}) 2024-09-15T04:00:52.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.592+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.592+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3400636892}) 2024-09-15T04:00:52.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.592+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1892861747}) 2024-09-15T04:00:52.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157464768}) 2024-09-15T04:00:52.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=965758945}) 2024-09-15T04:00:52.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157464768}) 2024-09-15T04:00:52.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3414632954}) 2024-09-15T04:00:52.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f9068709640 1 -- 172.21.15.110:0/2495896985 >> v2:172.21.15.177:6812/28201 conn(0x7f9044038700 msgr2=0x7f904403abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f9068709640 1 --2- 172.21.15.110:0/2495896985 >> v2:172.21.15.177:6812/28201 conn(0x7f9044038700 0x7f904403abb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9060068040 tx=0x7f905c032000 comp rx=0 tx=0).stop 2024-09-15T04:00:52.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f9068709640 10 monclient: shutdown 2024-09-15T04:00:52.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f9068709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:52.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f9068709640 1 -- 172.21.15.110:0/2495896985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 msgr2=0x7f906009a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:52.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f9068709640 1 --2- 172.21.15.110:0/2495896985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 0x7f906009a580 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f905800ceb0 tx=0x7f9058007870 comp rx=0 tx=0).stop 2024-09-15T04:00:52.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f9068709640 1 -- 172.21.15.110:0/2495896985 shutdown_connections 2024-09-15T04:00:52.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f9068709640 1 --2- 172.21.15.110:0/2495896985 >> v2:172.21.15.177:6812/28201 conn(0x7f9044038700 0x7f904403abb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f9068709640 1 --2- 172.21.15.110:0/2495896985 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f90600a1a60 0x7f906009a580 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f9068709640 1 --2- 172.21.15.110:0/2495896985 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f90600a0350 0x7f906009a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f9068709640 1 --2- 172.21.15.110:0/2495896985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f906009f870 0x7f9060099b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:52.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.593+0000 7f9068709640 1 -- 172.21.15.110:0/2495896985 >> 172.21.15.110:0/2495896985 conn(0x7f9060063270 msgr2=0x7f906009b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:52.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.594+0000 7f9068709640 1 -- 172.21.15.110:0/2495896985 shutdown_connections 2024-09-15T04:00:52.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:52.594+0000 7f9068709640 1 -- 172.21.15.110:0/2495896985 wait complete. 2024-09-15T04:00:52.599 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140122401}) 2024-09-15T04:00:52.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3230674593}) 2024-09-15T04:00:52.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140122401}) 2024-09-15T04:00:52.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404375276}) 2024-09-15T04:00:52.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.595+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.595+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108719819}) 2024-09-15T04:00:52.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.595+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1130961534}) 2024-09-15T04:00:52.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.595+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.595+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108719819}) 2024-09-15T04:00:52.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.595+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4110768553}) 2024-09-15T04:00:52.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.595+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=1} crc {logm=1145926966,mdsmap=1531151339,mgr=4138502149,mgr_command_descs=3545737492,mgr_metadata=3491211682,mgrstat=2814539991,mon_config_key=551709721}) 2024-09-15T04:00:52.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=82,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=6} crc {logm=3869061470,mdsmap=1531151339,mgr=4138502149,mgr_command_descs=3545737492,mgrstat=3670471458}) 2024-09-15T04:00:52.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=1} crc {logm=1145926966,mdsmap=1531151339,mgr=4138502149,mgr_command_descs=3545737492,mgr_metadata=3491211682,mgrstat=2814539991,mon_config_key=551709721}) 2024-09-15T04:00:52.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=2} crc {logm=3288745392,mdsmap=1531151339,mgr=4138502149,mgr_command_descs=3545737492,mgr_metadata=3491211682,mgrstat=1925984991,mon_config_key=3230398759}) 2024-09-15T04:00:52.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=84} crc {mon_config_key=2267758329,monmap=3538901833,osd_metadata=3281624269,osd_pg_creating=3219494227,osd_snap=3178633393,osdmap=2546206280}) 2024-09-15T04:00:52.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=84} crc {mon_config_key=2267758329,monmap=3538901833,osd_metadata=3281624269,osd_pg_creating=3219494227,osd_snap=3178633393,osdmap=622681323}) 2024-09-15T04:00:52.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=84} crc {mon_config_key=2267758329,monmap=3538901833,osd_metadata=3281624269,osd_pg_creating=3219494227,osd_snap=3178633393,osdmap=2546206280}) 2024-09-15T04:00:52.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=85} crc {mon_config_key=2267758329,monmap=1680381720,osd_metadata=3281624269,osd_pg_creating=3219494227,osd_snap=3178633393,osdmap=3670595481}) 2024-09-15T04:00:52.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.597+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.597+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2406396024}) 2024-09-15T04:00:52.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.597+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1109285696}) 2024-09-15T04:00:52.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.597+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.598+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2406396024}) 2024-09-15T04:00:52.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.598+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1252821392}) 2024-09-15T04:00:52.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.598+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.598+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=3} crc {osdmap=2969065485}) 2024-09-15T04:00:52.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.598+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=3} crc {osdmap=1977916817}) 2024-09-15T04:00:52.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.598+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:00:52.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.598+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=3} crc {osdmap=2969065485}) 2024-09-15T04:00:52.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:00:52.598+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=4} crc {osdmap=2524640270}) 2024-09-15T04:00:52.605 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T04:00:53.606 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T04:00:53.606 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:00:53.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.703+0000 7fb26651d640 10 monclient: get_monmap_and_config 2024-09-15T04:00:53.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.703+0000 7fb26651d640 10 monclient: build_initial_monmap 2024-09-15T04:00:53.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.703+0000 7fb26651d640 10 monclient: monmap: 2024-09-15T04:00:53.704 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:53.705 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:53.705 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:00:53.704405+0000 2024-09-15T04:00:53.705 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:00:53.704405+0000 2024-09-15T04:00:53.705 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:53.705 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:53.705 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:00:53.705 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:00:53.705 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:00:53.705 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:00:53.705 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:00:53.705 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 1 Processor -- start 2024-09-15T04:00:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 1 -- start start 2024-09-15T04:00:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 10 monclient: init 2024-09-15T04:00:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-15T04:00:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb260131ac0 0x7fb260131e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 10 monclient(hunting): picked mon.noname-b con 0x7fb260131ac0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb260097150 addr v1:172.21.15.177:6789/0 2024-09-15T04:00:53.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb26009bfd0 addr v1:172.21.15.110:6789/0 2024-09-15T04:00:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb260095ec0 con 0x7fb26009bfd0 2024-09-15T04:00:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26009cbd0 con 0x7fb260097150 2024-09-15T04:00:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb260095be0 con 0x7fb260131ac0 2024-09-15T04:00:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb26651d640 10 monclient(hunting): authenticate will time out at 877.348780s 2024-09-15T04:00:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb25ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb260131ac0 0x7fb260131e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb264aba640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb26009bfd0 0x7fb2601399b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51202/0 (socket says 172.21.15.110:51202) 2024-09-15T04:00:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb264aba640 1 -- 172.21.15.110:0/2055663017 learned_addr learned my addr 172.21.15.110:0/2055663017 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:00:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb25ffff640 10 monclient(hunting): get_auth_request con 0x7fb260131ac0 auth_method 0 2024-09-15T04:00:53.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb25ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb25ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb25ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb25ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.704+0000 7fb25ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.705+0000 7fb25ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.705+0000 7fb25ffff640 10 monclient(hunting): handle_auth_done global_id 4835 payload 274 2024-09-15T04:00:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.705+0000 7fb25ffff640 1 -- 172.21.15.110:0/2055663017 >> v1:172.21.15.177:6789/0 conn(0x7fb260097150 legacy=0x7fb26009b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.705+0000 7fb25ffff640 1 -- 172.21.15.110:0/2055663017 >> v1:172.21.15.110:6789/0 conn(0x7fb26009bfd0 legacy=0x7fb2601399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.705+0000 7fb25ffff640 10 monclient: _finish_hunting 0 2024-09-15T04:00:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.705+0000 7fb25ffff640 1 monclient: found mon.noname-b 2024-09-15T04:00:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.705+0000 7fb25ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.705+0000 7fb25ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:00:53.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.705+0000 7fb25ffff640 1 -- 172.21.15.110:0/2055663017 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2600961a0 con 0x7fb260131ac0 2024-09-15T04:00:53.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.707+0000 7fb25ffff640 1 -- 172.21.15.110:0/2055663017 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb260131ac0 msgr2=0x7fb260131e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:53.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.707+0000 7fb25ffff640 1 --2- 172.21.15.110:0/2055663017 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb260131ac0 0x7fb260131e90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb250009880 tx=0x7fb2500090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cb0d8bc2a093633 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:53.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb25effd640 1 -- 172.21.15.110:0/2055663017 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250032440 con 0x7fb260131ac0 2024-09-15T04:00:53.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb25effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:53.716 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb25effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:00:53.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb25effd640 10 monclient: dump: 2024-09-15T04:00:53.717 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:53.717 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:53.717 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:53.717 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:53.717 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:53.717 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:53.717 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:53.717 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:53.717 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:53.717 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:53.717 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb25effd640 10 monclient: _finish_auth 0 2024-09-15T04:00:53.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb25effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:53.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb25effd640 1 -- 172.21.15.110:0/2055663017 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2500325a0 con 0x7fb260131ac0 2024-09-15T04:00:53.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb26651d640 5 monclient: authenticate success, global_id 4835 2024-09-15T04:00:53.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb26651d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:53.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb25effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:53.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb26651d640 10 monclient: get_monmap_and_config success 2024-09-15T04:00:53.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb26651d640 4 set_mon_vals no callback set 2024-09-15T04:00:53.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb25effd640 1 -- 172.21.15.110:0/2055663017 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250032940 con 0x7fb260131ac0 2024-09-15T04:00:53.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb25effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:53.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb25effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:53.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb25effd640 10 monclient: dump: 2024-09-15T04:00:53.718 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:53.718 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:53.719 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:53.719 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:53.719 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:53.719 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:53.719 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:53.719 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:53.719 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:53.719 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:53.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb26651d640 10 monclient: shutdown 2024-09-15T04:00:53.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb26651d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:53.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb26651d640 1 -- 172.21.15.110:0/2055663017 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb260131ac0 msgr2=0x7fb260131e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:53.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb26651d640 1 --2- 172.21.15.110:0/2055663017 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb260131ac0 0x7fb260131e90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb250009880 tx=0x7fb2500090f0 comp rx=0 tx=0).stop 2024-09-15T04:00:53.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb26651d640 1 -- 172.21.15.110:0/2055663017 shutdown_connections 2024-09-15T04:00:53.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb26651d640 1 --2- 172.21.15.110:0/2055663017 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb260131ac0 0x7fb260131e90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:53.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb26651d640 1 -- 172.21.15.110:0/2055663017 >> 172.21.15.110:0/2055663017 conn(0x7fb2600a1440 msgr2=0x7fb2600a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:53.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb26651d640 1 -- 172.21.15.110:0/2055663017 shutdown_connections 2024-09-15T04:00:53.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.708+0000 7fb26651d640 1 -- 172.21.15.110:0/2055663017 wait complete. 2024-09-15T04:00:53.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.709+0000 7fb26651d640 10 monclient: build_initial_monmap 2024-09-15T04:00:53.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.709+0000 7fb26651d640 10 monclient: monmap: 2024-09-15T04:00:53.720 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:53.720 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:00:53.720 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:00:53.720 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:00:53.720 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:53.720 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:53.720 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:00:53.721 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:00:53.721 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:00:53.721 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:53.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.709+0000 7fb26651d640 1 Processor -- start 2024-09-15T04:00:53.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.709+0000 7fb26651d640 1 -- start start 2024-09-15T04:00:53.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.709+0000 7fb26651d640 10 monclient: init 2024-09-15T04:00:53.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.709+0000 7fb26651d640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:53.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.709+0000 7fb26651d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T04:00:53.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.709+0000 7fb26651d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb260097150 0x7fb260140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:53.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb26651d640 10 monclient(hunting): picked mon.noname-a con 0x7fb260097150 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:53.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb26651d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb26009bfd0 0x7fb260130a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:53.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb26651d640 10 monclient(hunting): picked mon.noname-c con 0x7fb26009bfd0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:53.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb26651d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb260131ac0 0x7fb26011a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:53.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb26651d640 10 monclient(hunting): picked mon.noname-b con 0x7fb260131ac0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:53.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb26651d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:53.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb26651d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26011db50 con 0x7fb260097150 2024-09-15T04:00:53.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb26651d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:53.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb26651d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26011c5a0 con 0x7fb260131ac0 2024-09-15T04:00:53.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb26651d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:53.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb26651d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26009d560 con 0x7fb26009bfd0 2024-09-15T04:00:53.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb26651d640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:53.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb26651d640 10 monclient(hunting): authenticate will time out at 877.354050s 2024-09-15T04:00:53.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb260097150 0x7fb260140eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:53.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb260097150 0x7fb260140eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54002/0 (socket says 172.21.15.110:54002) 2024-09-15T04:00:53.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 1 -- 172.21.15.110:0/1516605697 learned_addr learned my addr 172.21.15.110:0/1516605697 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:53.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 10 monclient(hunting): get_auth_request con 0x7fb260097150 auth_method 0 2024-09-15T04:00:53.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:53.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:53.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:53.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25f7fe640 1 --2- 172.21.15.110:0/1516605697 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb26009bfd0 0x7fb260130a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:53.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25f7fe640 10 monclient(hunting): get_auth_request con 0x7fb26009bfd0 auth_method 0 2024-09-15T04:00:53.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:53.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:53.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:53.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:53.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:53.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:53.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:53.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:53.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:53.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 10 monclient(hunting): handle_auth_done global_id 54144 payload 824 2024-09-15T04:00:53.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 1 -- 172.21.15.110:0/1516605697 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb26009bfd0 msgr2=0x7fb260130a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:53.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 1 --2- 172.21.15.110:0/1516605697 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb26009bfd0 0x7fb260130a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:53.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 1 -- 172.21.15.110:0/1516605697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb260131ac0 msgr2=0x7fb26011a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:53.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 1 --2- 172.21.15.110:0/1516605697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb260131ac0 0x7fb26011a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:53.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 10 monclient: _finish_hunting 0 2024-09-15T04:00:53.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 1 monclient: found mon.noname-a 2024-09-15T04:00:53.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:53.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:00:53.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25ffff640 1 -- 172.21.15.110:0/1516605697 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb250009390 con 0x7fb260097150 2024-09-15T04:00:53.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.710+0000 7fb25f7fe640 1 --2- 172.21.15.110:0/1516605697 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb26009bfd0 0x7fb260130a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:00:53.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb25ffff640 1 -- 172.21.15.110:0/1516605697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb260097150 msgr2=0x7fb260140eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:53.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb25ffff640 1 --2- 172.21.15.110:0/1516605697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb260097150 0x7fb260140eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb250009880 tx=0x7fb250032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:53.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb247fff640 1 -- 172.21.15.110:0/1516605697 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250031990 con 0x7fb260097150 2024-09-15T04:00:53.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb247fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:53.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb247fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:00:53.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb247fff640 10 monclient: dump: 2024-09-15T04:00:53.725 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:53.725 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:53.725 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:53.725 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:53.725 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:53.725 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:53.725 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:53.725 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:53.725 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:53.725 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:53.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb247fff640 10 monclient: _finish_auth 0 2024-09-15T04:00:53.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb247fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:53.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb247fff640 1 -- 172.21.15.110:0/1516605697 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb250031af0 con 0x7fb260097150 2024-09-15T04:00:53.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb26651d640 5 monclient: authenticate success, global_id 54144 2024-09-15T04:00:53.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb26651d640 10 monclient: _renew_subs 2024-09-15T04:00:53.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb26651d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:53.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb26651d640 1 -- 172.21.15.110:0/1516605697 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26009a7d0 con 0x7fb260097150 2024-09-15T04:00:53.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb247fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:53.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb247fff640 1 -- 172.21.15.110:0/1516605697 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb250031e70 con 0x7fb260097150 2024-09-15T04:00:53.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb247fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:53.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb247fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:53.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb247fff640 10 monclient: dump: 2024-09-15T04:00:53.726 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:53.726 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:53.727 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:53.727 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:53.727 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:53.727 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:53.727 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:53.727 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:53.727 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:53.727 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:53.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb25e7fc640 4 set_mon_vals no callback set 2024-09-15T04:00:53.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb26651d640 10 monclient: _renew_subs 2024-09-15T04:00:53.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb26651d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:53.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.712+0000 7fb26651d640 1 -- 172.21.15.110:0/1516605697 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26011cbf0 con 0x7fb260097150 2024-09-15T04:00:53.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.713+0000 7fb247fff640 1 -- 172.21.15.110:0/1516605697 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fb250039070 con 0x7fb260097150 2024-09-15T04:00:53.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.713+0000 7fb26651d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:53.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.713+0000 7fb26651d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:53.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.713+0000 7fb26651d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:53.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.713+0000 7fb26651d640 1 -- 172.21.15.110:0/1516605697 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb260070950 con 0x7fb260097150 2024-09-15T04:00:53.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.713+0000 7fb247fff640 1 --2- 172.21.15.110:0/1516605697 >> v2:172.21.15.177:6812/28201 conn(0x7fb24c038700 0x7fb24c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:53.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.713+0000 7fb247fff640 1 -- 172.21.15.110:0/1516605697 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fb25006c190 con 0x7fb260097150 2024-09-15T04:00:53.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.714+0000 7fb25f7fe640 1 --2- 172.21.15.110:0/1516605697 >> v2:172.21.15.177:6812/28201 conn(0x7fb24c038700 0x7fb24c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:53.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.714+0000 7fb25f7fe640 10 monclient: get_auth_request con 0x7fb24c038700 auth_method 0 2024-09-15T04:00:53.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.716+0000 7fb25f7fe640 1 --2- 172.21.15.110:0/1516605697 >> v2:172.21.15.177:6812/28201 conn(0x7fb24c038700 0x7fb24c03abb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb254006fd0 tx=0x7fb254009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:53.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.718+0000 7fb247fff640 1 -- 172.21.15.110:0/1516605697 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb25003c620 con 0x7fb260097150 2024-09-15T04:00:53.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.718+0000 7fb247fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:53.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.718+0000 7fb247fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:00:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.930+0000 7fb26651d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:00:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.930+0000 7fb26651d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:00:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.930+0000 7fb26651d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:53.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.930+0000 7fb26651d640 1 -- 172.21.15.110:0/1516605697 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb260070950 con 0x7fb260097150 2024-09-15T04:00:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.931+0000 7fb247fff640 1 -- 172.21.15.110:0/1516605697 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fb250040020 con 0x7fb260097150 2024-09-15T04:00:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.931+0000 7fb247fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:00:53.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.931+0000 7fb247fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:00:53.932 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:00:53.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.933+0000 7fb26651d640 1 -- 172.21.15.110:0/1516605697 >> v2:172.21.15.177:6812/28201 conn(0x7fb24c038700 msgr2=0x7fb24c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:53.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.933+0000 7fb26651d640 1 --2- 172.21.15.110:0/1516605697 >> v2:172.21.15.177:6812/28201 conn(0x7fb24c038700 0x7fb24c03abb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb254006fd0 tx=0x7fb254009060 comp rx=0 tx=0).stop 2024-09-15T04:00:53.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.933+0000 7fb26651d640 10 monclient: shutdown 2024-09-15T04:00:53.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.933+0000 7fb26651d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:53.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.933+0000 7fb26651d640 1 -- 172.21.15.110:0/1516605697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb260097150 msgr2=0x7fb260140eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.933+0000 7fb26651d640 1 --2- 172.21.15.110:0/1516605697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb260097150 0x7fb260140eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb250009880 tx=0x7fb250032000 comp rx=0 tx=0).stop 2024-09-15T04:00:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.933+0000 7fb26651d640 1 -- 172.21.15.110:0/1516605697 shutdown_connections 2024-09-15T04:00:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.933+0000 7fb26651d640 1 --2- 172.21.15.110:0/1516605697 >> v2:172.21.15.177:6812/28201 conn(0x7fb24c038700 0x7fb24c03abb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.933+0000 7fb26651d640 1 --2- 172.21.15.110:0/1516605697 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb260131ac0 0x7fb26011a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.933+0000 7fb26651d640 1 --2- 172.21.15.110:0/1516605697 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb26009bfd0 0x7fb260130a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.933+0000 7fb26651d640 1 --2- 172.21.15.110:0/1516605697 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb260097150 0x7fb260140eb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.933+0000 7fb26651d640 1 -- 172.21.15.110:0/1516605697 >> 172.21.15.110:0/1516605697 conn(0x7fb2600a1440 msgr2=0x7fb26009fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.933+0000 7fb26651d640 1 -- 172.21.15.110:0/1516605697 shutdown_connections 2024-09-15T04:00:53.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:53.933+0000 7fb26651d640 1 -- 172.21.15.110:0/1516605697 wait complete. 2024-09-15T04:00:53.944 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T04:00:53.944 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T04:00:53.944 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T04:00:53.944 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T04:00:54.037 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.036+0000 7fce10309640 10 monclient: get_monmap_and_config 2024-09-15T04:00:54.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.036+0000 7fce10309640 10 monclient: build_initial_monmap 2024-09-15T04:00:54.038 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.036+0000 7fce10309640 10 monclient: monmap: 2024-09-15T04:00:54.038 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:54.038 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.038 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:00:54.037957+0000 2024-09-15T04:00:54.038 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:00:54.037957+0000 2024-09-15T04:00:54.038 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:54.038 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.038 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:00:54.038 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:00:54.038 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:00:54.039 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:00:54.039 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:00:54.039 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.037+0000 7fce10309640 1 Processor -- start 2024-09-15T04:00:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.037+0000 7fce10309640 1 -- start start 2024-09-15T04:00:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.037+0000 7fce10309640 10 monclient: init 2024-09-15T04:00:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce10309640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:54.049 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce10309640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-15T04:00:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce10309640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce080a1cf0 0x7fce0809e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce10309640 10 monclient(hunting): picked mon.noname-b con 0x7fce080a1cf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce10309640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce08131a80 addr v1:172.21.15.177:6789/0 2024-09-15T04:00:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce10309640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fce08136940 0x7fce08138d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce10309640 10 monclient(hunting): picked mon.noname-a con 0x7fce08136940 addr v2:172.21.15.110:3300/0 2024-09-15T04:00:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce10309640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce0809cb70 con 0x7fce08131a80 2024-09-15T04:00:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce10309640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce10309640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce080a20c0 con 0x7fce08136940 2024-09-15T04:00:54.050 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce10309640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce10309640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0809c330 con 0x7fce080a1cf0 2024-09-15T04:00:54.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce10309640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:54.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce10309640 10 monclient(hunting): authenticate will time out at 877.682334s 2024-09-15T04:00:54.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce080a1cf0 0x7fce0809e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e8a6640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fce08136940 0x7fce08138d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce080a1cf0 0x7fce0809e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50766/0 (socket says 172.21.15.110:50766) 2024-09-15T04:00:54.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 1 -- 172.21.15.110:0/2858759758 learned_addr learned my addr 172.21.15.110:0/2858759758 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:54.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 10 monclient(hunting): get_auth_request con 0x7fce080a1cf0 auth_method 0 2024-09-15T04:00:54.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e8a6640 10 monclient(hunting): get_auth_request con 0x7fce08136940 auth_method 0 2024-09-15T04:00:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e8a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e8a6640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e8a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e8a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e8a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e8a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 10 monclient(hunting): handle_auth_done global_id 4841 payload 274 2024-09-15T04:00:54.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 1 -- 172.21.15.110:0/2858759758 >> v2:172.21.15.110:3300/0 conn(0x7fce08136940 msgr2=0x7fce08138d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 1 --2- 172.21.15.110:0/2858759758 >> v2:172.21.15.110:3300/0 conn(0x7fce08136940 0x7fce08138d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 1 -- 172.21.15.110:0/2858759758 >> v1:172.21.15.177:6789/0 conn(0x7fce08131a80 legacy=0x7fce0809eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 10 monclient: _finish_hunting 0 2024-09-15T04:00:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 1 monclient: found mon.noname-b 2024-09-15T04:00:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:00:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e0a5640 1 -- 172.21.15.110:0/2858759758 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce08131fb0 con 0x7fce080a1cf0 2024-09-15T04:00:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.038+0000 7fce0e8a6640 1 --2- 172.21.15.110:0/2858759758 >> v2:172.21.15.110:3300/0 conn(0x7fce08136940 0x7fce08138d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:00:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce0e0a5640 1 -- 172.21.15.110:0/2858759758 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce080a1cf0 msgr2=0x7fce0809e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce0e0a5640 1 --2- 172.21.15.110:0/2858759758 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce080a1cf0 0x7fce0809e660 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce081321a0 tx=0x7fce04009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67d5011883238d55 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce0d0a3640 1 -- 172.21.15.110:0/2858759758 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce04007610 con 0x7fce080a1cf0 2024-09-15T04:00:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce0d0a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce0d0a3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:00:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce0d0a3640 10 monclient: dump: 2024-09-15T04:00:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.054 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce0d0a3640 10 monclient: _finish_auth 0 2024-09-15T04:00:54.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce0d0a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce0d0a3640 1 -- 172.21.15.110:0/2858759758 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0402d050 con 0x7fce080a1cf0 2024-09-15T04:00:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce10309640 5 monclient: authenticate success, global_id 4841 2024-09-15T04:00:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce10309640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce0d0a3640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce10309640 10 monclient: get_monmap_and_config success 2024-09-15T04:00:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce0d0a3640 1 -- 172.21.15.110:0/2858759758 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce040079e0 con 0x7fce080a1cf0 2024-09-15T04:00:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce0d0a3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce10309640 4 set_mon_vals no callback set 2024-09-15T04:00:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce0d0a3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.041+0000 7fce0d0a3640 10 monclient: dump: 2024-09-15T04:00:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.055 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.056 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.056 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.056 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.056 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.056 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.056 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.056 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.056 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.042+0000 7fce10309640 10 monclient: shutdown 2024-09-15T04:00:54.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.042+0000 7fce10309640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:54.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.042+0000 7fce10309640 1 -- 172.21.15.110:0/2858759758 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce080a1cf0 msgr2=0x7fce0809e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.042+0000 7fce10309640 1 --2- 172.21.15.110:0/2858759758 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce080a1cf0 0x7fce0809e660 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce081321a0 tx=0x7fce04009840 comp rx=0 tx=0).stop 2024-09-15T04:00:54.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.042+0000 7fce10309640 1 -- 172.21.15.110:0/2858759758 shutdown_connections 2024-09-15T04:00:54.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.042+0000 7fce10309640 1 --2- 172.21.15.110:0/2858759758 >> v2:172.21.15.110:3300/0 conn(0x7fce08136940 0x7fce08138d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.042+0000 7fce10309640 1 --2- 172.21.15.110:0/2858759758 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce080a1cf0 0x7fce0809e660 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.042+0000 7fce10309640 1 -- 172.21.15.110:0/2858759758 >> 172.21.15.110:0/2858759758 conn(0x7fce08062f70 msgr2=0x7fce08063340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:54.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.042+0000 7fce10309640 1 -- 172.21.15.110:0/2858759758 shutdown_connections 2024-09-15T04:00:54.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.042+0000 7fce10309640 1 -- 172.21.15.110:0/2858759758 wait complete. 2024-09-15T04:00:54.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.042+0000 7fce10309640 10 monclient: build_initial_monmap 2024-09-15T04:00:54.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.042+0000 7fce10309640 10 monclient: monmap: 2024-09-15T04:00:54.057 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:54.057 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:00:54.057 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:00:54.058 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:00:54.058 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:54.058 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.058 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:00:54.058 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:00:54.058 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:00:54.058 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 1 Processor -- start 2024-09-15T04:00:54.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 1 -- start start 2024-09-15T04:00:54.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 10 monclient: init 2024-09-15T04:00:54.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:54.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T04:00:54.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce080a1cf0 0x7fce080975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 10 monclient(hunting): picked mon.noname-a con 0x7fce080a1cf0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:54.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce08131a80 0x7fce08097ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 10 monclient(hunting): picked mon.noname-c con 0x7fce08131a80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:54.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce08136940 0x7fce08098020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 10 monclient(hunting): picked mon.noname-b con 0x7fce08136940 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:54.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce080a11e0 con 0x7fce080a1cf0 2024-09-15T04:00:54.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce080a5710 con 0x7fce08136940 2024-09-15T04:00:54.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0809a940 con 0x7fce08131a80 2024-09-15T04:00:54.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:54.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce10309640 10 monclient(hunting): authenticate will time out at 877.687611s 2024-09-15T04:00:54.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce080a1cf0 0x7fce080975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce080a1cf0 0x7fce080975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54028/0 (socket says 172.21.15.110:54028) 2024-09-15T04:00:54.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 1 -- 172.21.15.110:0/3217226865 learned_addr learned my addr 172.21.15.110:0/3217226865 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:54.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 10 monclient(hunting): get_auth_request con 0x7fce080a1cf0 auth_method 0 2024-09-15T04:00:54.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0d8a4640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce08131a80 0x7fce08097ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0d8a4640 10 monclient(hunting): get_auth_request con 0x7fce08131a80 auth_method 0 2024-09-15T04:00:54.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0d8a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0d8a4640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0d8a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0d8a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0d8a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e8a6640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce08136940 0x7fce08098020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0d8a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e8a6640 10 monclient(hunting): get_auth_request con 0x7fce08136940 auth_method 0 2024-09-15T04:00:54.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e8a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e8a6640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e8a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 10 monclient(hunting): handle_auth_done global_id 54150 payload 824 2024-09-15T04:00:54.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 1 -- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce08131a80 msgr2=0x7fce08097ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce08131a80 0x7fce08097ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 1 -- 172.21.15.110:0/3217226865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce08136940 msgr2=0x7fce08098020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce08136940 0x7fce08098020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 10 monclient: _finish_hunting 0 2024-09-15T04:00:54.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 1 monclient: found mon.noname-a 2024-09-15T04:00:54.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:54.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:00:54.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e0a5640 1 -- 172.21.15.110:0/3217226865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce00009390 con 0x7fce080a1cf0 2024-09-15T04:00:54.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0d8a4640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fce08131a80 0x7fce08097ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:00:54.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.043+0000 7fce0e8a6640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fce08136940 0x7fce08098020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T04:00:54.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fce0e0a5640 1 -- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce080a1cf0 msgr2=0x7fce080975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fce0e0a5640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce080a1cf0 0x7fce080975a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fce04009f50 tx=0x7fce04036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fcdfe7fc640 1 -- 172.21.15.110:0/3217226865 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0403b830 con 0x7fce080a1cf0 2024-09-15T04:00:54.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fcdfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fcdfe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:00:54.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fcdfe7fc640 10 monclient: dump: 2024-09-15T04:00:54.063 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.063 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.063 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.063 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.063 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.063 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.063 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.064 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.064 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.064 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fcdfe7fc640 10 monclient: _finish_auth 0 2024-09-15T04:00:54.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fcdfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:54.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fce10309640 5 monclient: authenticate success, global_id 54150 2024-09-15T04:00:54.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fcdfe7fc640 1 -- 172.21.15.110:0/3217226865 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0403b990 con 0x7fce080a1cf0 2024-09-15T04:00:54.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fcdfe7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:54.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fcdfe7fc640 1 -- 172.21.15.110:0/3217226865 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0403bd50 con 0x7fce080a1cf0 2024-09-15T04:00:54.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fce10309640 10 monclient: _renew_subs 2024-09-15T04:00:54.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fce10309640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:54.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fce10309640 1 -- 172.21.15.110:0/3217226865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce04009ae0 con 0x7fce080a1cf0 2024-09-15T04:00:54.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fcdfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fcdfe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:54.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fce0c8a2640 4 set_mon_vals no callback set 2024-09-15T04:00:54.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fcdfe7fc640 10 monclient: dump: 2024-09-15T04:00:54.065 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.065 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.065 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.065 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.065 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.065 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.065 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.065 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.065 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.065 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fce10309640 10 monclient: _renew_subs 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fce10309640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.046+0000 7fce10309640 1 -- 172.21.15.110:0/3217226865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce081312d0 con 0x7fce080a1cf0 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fcdfe7fc640 1 -- 172.21.15.110:0/3217226865 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fce04038070 con 0x7fce080a1cf0 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce10309640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce10309640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce08098560 0x7fce08136d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce10309640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce10309640 1 -- 172.21.15.110:0/3217226865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0806f1e0 con 0x7fce08098560 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce0d8a4640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce08098560 0x7fce08136d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce0d8a4640 10 monclient: get_auth_request con 0x7fce08098560 auth_method 0 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce0d8a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce0d8a4640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce0d8a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fcdfe7fc640 1 --2- 172.21.15.110:0/3217226865 >> v2:172.21.15.177:6812/28201 conn(0x7fcdec038700 0x7fcdec03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce0d8a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.066 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce0d8a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce0d8a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fcdfe7fc640 1 -- 172.21.15.110:0/3217226865 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fce0402ae40 con 0x7fce080a1cf0 2024-09-15T04:00:54.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce0d8a4640 10 monclient(hunting): handle_auth_done global_id 54153 payload 824 2024-09-15T04:00:54.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce0d8a4640 1 -- 172.21.15.110:0/3217226865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce0809ab10 con 0x7fce08098560 2024-09-15T04:00:54.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce0e8a6640 1 --2- 172.21.15.110:0/3217226865 >> v2:172.21.15.177:6812/28201 conn(0x7fcdec038700 0x7fcdec03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.047+0000 7fce0e8a6640 10 monclient: get_auth_request con 0x7fcdec038700 auth_method 0 2024-09-15T04:00:54.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.049+0000 7fce0d8a4640 1 -- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce08098560 msgr2=0x7fce08136d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.069 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.049+0000 7fce0d8a4640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce08098560 0x7fce08136d10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcdf800fa50 tx=0x7fcdf8007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.049+0000 7fcdfe7fc640 1 -- 172.21.15.110:0/3217226865 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf800dda0 con 0x7fce08098560 2024-09-15T04:00:54.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.049+0000 7fcdfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.049+0000 7fcdfe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:54.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.049+0000 7fcdfe7fc640 10 monclient: dump: 2024-09-15T04:00:54.070 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.070 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.070 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.070 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.070 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.070 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.070 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.071 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.071 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.071 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.050+0000 7fcdfe7fc640 1 -- 172.21.15.110:0/3217226865 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fcdf8014470 con 0x7fce08098560 2024-09-15T04:00:54.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.050+0000 7fcdfe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:54.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.050+0000 7fcdfe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:00:54.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.050+0000 7fcdfe7fc640 1 -- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce08098560 msgr2=0x7fce08136d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.050+0000 7fcdfe7fc640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce08098560 0x7fce08136d10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcdf800fa50 tx=0x7fcdf8007e10 comp rx=0 tx=0).stop 2024-09-15T04:00:54.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.050+0000 7fcdfe7fc640 1 -- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce08098560 msgr2=0x7fce08136d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:00:54.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.050+0000 7fcdfe7fc640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce08098560 0x7fce08136d10 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcdf800fa50 tx=0x7fcdf8007e10 comp rx=0 tx=0).stop 2024-09-15T04:00:54.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.050+0000 7fce0e8a6640 1 --2- 172.21.15.110:0/3217226865 >> v2:172.21.15.177:6812/28201 conn(0x7fcdec038700 0x7fcdec03abb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fce00009360 tx=0x7fce00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.055+0000 7fce10309640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:00:54.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.055+0000 7fce10309640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce0809ade0 0x7fce08137250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.055+0000 7fce10309640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.055+0000 7fce10309640 1 -- 172.21.15.110:0/3217226865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce080647e0 con 0x7fce0809ade0 2024-09-15T04:00:54.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.055+0000 7fce0d8a4640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce0809ade0 0x7fce08137250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.056+0000 7fce0d8a4640 10 monclient: get_auth_request con 0x7fce0809ade0 auth_method 0 2024-09-15T04:00:54.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.056+0000 7fce0d8a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.056+0000 7fce0d8a4640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.056+0000 7fce0d8a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.056+0000 7fce0d8a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.056+0000 7fce0d8a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.056+0000 7fce0d8a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.056+0000 7fce0d8a4640 10 monclient(hunting): handle_auth_done global_id 54156 payload 824 2024-09-15T04:00:54.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.056+0000 7fce0d8a4640 1 -- 172.21.15.110:0/3217226865 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fce08067130 con 0x7fce0809ade0 2024-09-15T04:00:54.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.058+0000 7fce0d8a4640 1 -- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce0809ade0 msgr2=0x7fce08137250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.058+0000 7fce0d8a4640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce0809ade0 0x7fce08137250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8019000 tx=0x7fcdf801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.058+0000 7fcdfe7fc640 1 -- 172.21.15.110:0/3217226865 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf801ed00 con 0x7fce0809ade0 2024-09-15T04:00:54.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.058+0000 7fcdfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.058+0000 7fcdfe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:54.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.058+0000 7fcdfe7fc640 10 monclient: dump: 2024-09-15T04:00:54.073 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.073 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.073 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.073 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.074 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.074 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.074 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.074 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.074 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.074 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.058+0000 7fcdfe7fc640 1 -- 172.21.15.110:0/3217226865 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4409 (secure 0 0 0) 0x7fcdf8020410 con 0x7fce0809ade0 2024-09-15T04:00:54.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.058+0000 7fcdfe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:00:54.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.058+0000 7fcdfe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:00:54.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.058+0000 7fcdfe7fc640 1 -- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce0809ade0 msgr2=0x7fce08137250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.058+0000 7fcdfe7fc640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce0809ade0 0x7fce08137250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8019000 tx=0x7fcdf801a000 comp rx=0 tx=0).stop 2024-09-15T04:00:54.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.058+0000 7fcdfe7fc640 1 -- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce0809ade0 msgr2=0x7fce08137250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:00:54.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.058+0000 7fcdfe7fc640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce0809ade0 0x7fce08137250 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8019000 tx=0x7fcdf801a000 comp rx=0 tx=0).stop 2024-09-15T04:00:54.075 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:00:54.075 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:00:54.075 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:00:54.075 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "leader", 2024-09-15T04:00:54.075 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 22, 2024-09-15T04:00:54.076 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:00:54.076 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:00:54.076 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:00:54.076 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:00:54.076 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.076 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 4, 2024-09-15T04:00:54.076 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:00:54.076 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:00:54.076 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:00:54.076 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:54.077 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:54.077 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:54.077 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:54.077 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:54.077 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:54.077 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:54.077 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:54.077 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:54.077 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.077 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:00:54.077 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:00:54.077 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:54.077 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:54.078 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:54.078 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:54.078 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:54.078 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:54.078 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:54.078 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:54.078 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:54.078 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.078 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.078 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:00:54.078 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:00:54.078 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:00:54.079 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:00:54.079 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:00:54.079 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:00:54.079 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:00:54.079 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:00:54.079 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:00:54.079 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:00:54.079 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:00:54.079 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:00:54.079 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:00:54.079 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:00:54.079 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:00:54.079 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:00:54.080 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:00:54.080 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:54.080 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:54.080 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:54.080 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:54.080 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:54.080 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:54.080 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:54.080 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:54.080 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:54.080 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.080 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:00:54.080 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.080 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:00:54.081 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.081 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:00:54.081 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:00:54.081 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:54.081 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:54.081 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.081 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:54.081 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:00:54.081 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.081 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.081 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.082 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:54.082 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:00:54.082 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.082 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.082 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.082 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.082 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:00:54.082 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:00:54.082 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:54.082 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:54.082 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:54.082 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.082 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.082 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:00:54.083 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:00:54.083 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:54.083 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:54.083 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.083 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:54.083 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:00:54.083 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.083 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.083 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.083 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:54.083 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:00:54.083 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.083 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.083 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.084 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.084 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:00:54.084 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:00:54.084 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:54.084 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:54.084 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:54.084 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.084 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.084 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:00:54.084 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:00:54.084 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:54.084 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:54.084 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.084 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:54.085 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:00:54.085 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.085 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.085 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.085 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:54.085 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:00:54.085 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.085 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.085 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.085 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.085 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:00:54.085 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:00:54.085 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:54.086 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:54.086 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:54.086 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.086 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.086 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.086 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:00:54.086 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:00:54.086 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.086 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:54.086 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:54.086 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:00:54.086 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.087 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.087 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:00:54.087 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.087 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:54.087 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:54.087 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 3 2024-09-15T04:00:54.087 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.087 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.087 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.087 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:00:54.087 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:00:54.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.059+0000 7fce10309640 1 -- 172.21.15.110:0/3217226865 >> v2:172.21.15.177:6812/28201 conn(0x7fcdec038700 msgr2=0x7fcdec03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.059+0000 7fce10309640 1 --2- 172.21.15.110:0/3217226865 >> v2:172.21.15.177:6812/28201 conn(0x7fcdec038700 0x7fcdec03abb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fce00009360 tx=0x7fce00008040 comp rx=0 tx=0).stop 2024-09-15T04:00:54.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.059+0000 7fce10309640 10 monclient: shutdown 2024-09-15T04:00:54.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.059+0000 7fce10309640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:54.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.059+0000 7fce10309640 1 -- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce080a1cf0 msgr2=0x7fce080975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.059+0000 7fce10309640 1 --2- 172.21.15.110:0/3217226865 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fce080a1cf0 0x7fce080975a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fce04009f50 tx=0x7fce04036000 comp rx=0 tx=0).stop 2024-09-15T04:00:54.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.059+0000 7fce0e8a6640 1 -- 172.21.15.110:0/3217226865 reap_dead start 2024-09-15T04:00:54.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.059+0000 7fce0e8a6640 1 -- 172.21.15.110:0/3217226865 reap_dead start 2024-09-15T04:00:54.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.059+0000 7fce10309640 1 -- 172.21.15.110:0/3217226865 shutdown_connections 2024-09-15T04:00:54.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.059+0000 7fce10309640 1 -- 172.21.15.110:0/3217226865 >> 172.21.15.110:0/3217226865 conn(0x7fce08062f70 msgr2=0x7fce080a3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:54.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.059+0000 7fce10309640 1 -- 172.21.15.110:0/3217226865 shutdown_connections 2024-09-15T04:00:54.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.059+0000 7fce10309640 1 -- 172.21.15.110:0/3217226865 wait complete. 2024-09-15T04:00:54.090 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:00:54.090 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T04:00:54.090 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T04:00:54.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.182+0000 7f8b88a80640 10 monclient: get_monmap_and_config 2024-09-15T04:00:54.183 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.182+0000 7f8b88a80640 10 monclient: build_initial_monmap 2024-09-15T04:00:54.184 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.182+0000 7f8b88a80640 10 monclient: monmap: 2024-09-15T04:00:54.184 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:54.184 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.184 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:00:54.183732+0000 2024-09-15T04:00:54.184 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:00:54.183732+0000 2024-09-15T04:00:54.184 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:54.184 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.184 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:00:54.184 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:00:54.185 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:00:54.185 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:00:54.185 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:00:54.185 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.183+0000 7f8b88a80640 1 Processor -- start 2024-09-15T04:00:54.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.183+0000 7f8b88a80640 1 -- start start 2024-09-15T04:00:54.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.183+0000 7f8b88a80640 10 monclient: init 2024-09-15T04:00:54.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.183+0000 7f8b88a80640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:54.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.183+0000 7f8b88a80640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T04:00:54.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.183+0000 7f8b88a80640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8b840a1a10 addr v1:172.21.15.110:6789/0 2024-09-15T04:00:54.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.183+0000 7f8b88a80640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8b8409ee80 0x7f8b84131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.183+0000 7f8b88a80640 10 monclient(hunting): picked mon.noname-c con 0x7f8b8409ee80 addr v2:172.21.15.177:3300/0 2024-09-15T04:00:54.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.183+0000 7f8b88a80640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8b841369e0 0x7f8b84138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.183+0000 7f8b88a80640 10 monclient(hunting): picked mon.noname-a con 0x7f8b841369e0 addr v2:172.21.15.110:3300/0 2024-09-15T04:00:54.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b88a80640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8409f2c0 con 0x7f8b840a1a10 2024-09-15T04:00:54.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b88a80640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b88a80640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b840a0850 con 0x7f8b841369e0 2024-09-15T04:00:54.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b88a80640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b88a80640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8409c9a0 con 0x7f8b8409ee80 2024-09-15T04:00:54.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b88a80640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:54.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b88a80640 10 monclient(hunting): authenticate will time out at 877.828034s 2024-09-15T04:00:54.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b82d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8b841369e0 0x7f8b84138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b82d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8b841369e0 0x7f8b84138dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54040/0 (socket says 172.21.15.110:54040) 2024-09-15T04:00:54.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b82d9d640 1 -- 172.21.15.110:0/2398177838 learned_addr learned my addr 172.21.15.110:0/2398177838 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:54.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b82d9d640 10 monclient(hunting): get_auth_request con 0x7f8b841369e0 auth_method 0 2024-09-15T04:00:54.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b82d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 1 --2- 172.21.15.110:0/2398177838 >> v2:172.21.15.177:3300/0 conn(0x7f8b8409ee80 0x7f8b84131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 10 monclient(hunting): get_auth_request con 0x7f8b8409ee80 auth_method 0 2024-09-15T04:00:54.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 10 monclient(hunting): handle_auth_done global_id 4816 payload 274 2024-09-15T04:00:54.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 1 -- 172.21.15.110:0/2398177838 >> v2:172.21.15.110:3300/0 conn(0x7f8b841369e0 msgr2=0x7f8b84138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:54.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 1 --2- 172.21.15.110:0/2398177838 >> v2:172.21.15.110:3300/0 conn(0x7f8b841369e0 0x7f8b84138dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 1 -- 172.21.15.110:0/2398177838 >> v1:172.21.15.110:6789/0 conn(0x7f8b840a1a10 legacy=0x7f8b8409e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:54.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 10 monclient: _finish_hunting 0 2024-09-15T04:00:54.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 1 monclient: found mon.noname-c 2024-09-15T04:00:54.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:54.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T04:00:54.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.184+0000 7f8b81d9b640 1 -- 172.21.15.110:0/2398177838 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8409cc80 con 0x7f8b8409ee80 2024-09-15T04:00:54.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b81d9b640 1 -- 172.21.15.110:0/2398177838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8409ee80 msgr2=0x7f8b84131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b81d9b640 1 --2- 172.21.15.110:0/2398177838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8409ee80 0x7f8b84131b90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8b840a2790 tx=0x7f8b780090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f06afba43db4ad4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b8159a640 1 -- 172.21.15.110:0/2398177838 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7802ae00 con 0x7f8b8409ee80 2024-09-15T04:00:54.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b8159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:00:54.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b8159a640 10 monclient: dump: 2024-09-15T04:00:54.199 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.199 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.199 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.199 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.199 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.200 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.200 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.200 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.200 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.200 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b8159a640 10 monclient: _finish_auth 0 2024-09-15T04:00:54.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b8159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:54.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b8159a640 1 -- 172.21.15.110:0/2398177838 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b78032440 con 0x7f8b8409ee80 2024-09-15T04:00:54.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b88a80640 5 monclient: authenticate success, global_id 4816 2024-09-15T04:00:54.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b88a80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:54.200 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b8159a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b88a80640 10 monclient: get_monmap_and_config success 2024-09-15T04:00:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b88a80640 4 set_mon_vals no callback set 2024-09-15T04:00:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b8159a640 1 -- 172.21.15.110:0/2398177838 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b78032820 con 0x7f8b8409ee80 2024-09-15T04:00:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b8159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:00:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.187+0000 7f8b8159a640 10 monclient: dump: 2024-09-15T04:00:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.201 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.202 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.188+0000 7f8b88a80640 10 monclient: shutdown 2024-09-15T04:00:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.188+0000 7f8b88a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.188+0000 7f8b88a80640 1 -- 172.21.15.110:0/2398177838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8409ee80 msgr2=0x7f8b84131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.188+0000 7f8b88a80640 1 --2- 172.21.15.110:0/2398177838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8409ee80 0x7f8b84131b90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8b840a2790 tx=0x7f8b780090f0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.188+0000 7f8b88a80640 1 -- 172.21.15.110:0/2398177838 shutdown_connections 2024-09-15T04:00:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.188+0000 7f8b88a80640 1 --2- 172.21.15.110:0/2398177838 >> v2:172.21.15.110:3300/0 conn(0x7f8b841369e0 0x7f8b84138dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.188+0000 7f8b88a80640 1 --2- 172.21.15.110:0/2398177838 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b8409ee80 0x7f8b84131b90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.188+0000 7f8b88a80640 1 -- 172.21.15.110:0/2398177838 >> 172.21.15.110:0/2398177838 conn(0x7f8b8406e860 msgr2=0x7f8b8406ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:54.202 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.188+0000 7f8b88a80640 1 -- 172.21.15.110:0/2398177838 shutdown_connections 2024-09-15T04:00:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.188+0000 7f8b88a80640 1 -- 172.21.15.110:0/2398177838 wait complete. 2024-09-15T04:00:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.188+0000 7f8b88a80640 10 monclient: build_initial_monmap 2024-09-15T04:00:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.188+0000 7f8b88a80640 10 monclient: monmap: 2024-09-15T04:00:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:00:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:00:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:00:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:00:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:00:54.203 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:00:54.203 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 1 Processor -- start 2024-09-15T04:00:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 1 -- start start 2024-09-15T04:00:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 10 monclient: init 2024-09-15T04:00:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T04:00:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b8409ee80 0x7f8b8409b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 10 monclient(hunting): picked mon.noname-c con 0x7f8b8409ee80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b840a1a10 0x7f8b84097210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 10 monclient(hunting): picked mon.noname-b con 0x7f8b840a1a10 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b841369e0 0x7f8b84097750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 10 monclient(hunting): picked mon.noname-a con 0x7f8b841369e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8409b9d0 con 0x7f8b841369e0 2024-09-15T04:00:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b840a11c0 con 0x7f8b840a1a10 2024-09-15T04:00:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b840a0320 con 0x7f8b8409ee80 2024-09-15T04:00:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b88a80640 10 monclient(hunting): authenticate will time out at 877.833691s 2024-09-15T04:00:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b82d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b841369e0 0x7f8b84097750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b82d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b841369e0 0x7f8b84097750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54046/0 (socket says 172.21.15.110:54046) 2024-09-15T04:00:54.205 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b82d9d640 1 -- 172.21.15.110:0/3021348388 learned_addr learned my addr 172.21.15.110:0/3021348388 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:54.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b82d9d640 10 monclient(hunting): get_auth_request con 0x7f8b841369e0 auth_method 0 2024-09-15T04:00:54.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b82d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b8259c640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b8409ee80 0x7f8b8409b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b8259c640 10 monclient(hunting): get_auth_request con 0x7f8b8409ee80 auth_method 0 2024-09-15T04:00:54.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b8259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b8259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b81d9b640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b840a1a10 0x7f8b84097210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b82d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b82d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b82d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.206 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b81d9b640 10 monclient(hunting): get_auth_request con 0x7f8b840a1a10 auth_method 0 2024-09-15T04:00:54.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b81d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b81d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.189+0000 7f8b8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.190+0000 7f8b81d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.190+0000 7f8b81d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.190+0000 7f8b81d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.190+0000 7f8b82d9d640 10 monclient(hunting): handle_auth_done global_id 54162 payload 824 2024-09-15T04:00:54.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.190+0000 7f8b82d9d640 1 -- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b8409ee80 msgr2=0x7f8b8409b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.190+0000 7f8b82d9d640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b8409ee80 0x7f8b8409b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.190+0000 7f8b82d9d640 1 -- 172.21.15.110:0/3021348388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b840a1a10 msgr2=0x7f8b84097210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.207 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.190+0000 7f8b82d9d640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8b840a1a10 0x7f8b84097210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.190+0000 7f8b82d9d640 10 monclient: _finish_hunting 0 2024-09-15T04:00:54.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.190+0000 7f8b82d9d640 1 monclient: found mon.noname-a 2024-09-15T04:00:54.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.190+0000 7f8b82d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:54.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.190+0000 7f8b82d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:00:54.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.190+0000 7f8b82d9d640 1 -- 172.21.15.110:0/3021348388 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b78009390 con 0x7f8b841369e0 2024-09-15T04:00:54.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.190+0000 7f8b8259c640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b8409ee80 0x7f8b8409b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:00:54.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b82d9d640 1 -- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b841369e0 msgr2=0x7f8b84097750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b82d9d640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b841369e0 0x7f8b84097750 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c007c70 tx=0x7f8b7c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b727fc640 1 -- 172.21.15.110:0/3021348388 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c00dd50 con 0x7f8b841369e0 2024-09-15T04:00:54.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:00:54.208 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b727fc640 10 monclient: dump: 2024-09-15T04:00:54.208 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.209 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.209 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.209 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.209 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.209 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.209 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.209 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.209 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.209 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b727fc640 10 monclient: _finish_auth 0 2024-09-15T04:00:54.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:54.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b727fc640 1 -- 172.21.15.110:0/3021348388 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7c00f040 con 0x7f8b841369e0 2024-09-15T04:00:54.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:54.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b727fc640 1 -- 172.21.15.110:0/3021348388 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c013640 con 0x7f8b841369e0 2024-09-15T04:00:54.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b88a80640 5 monclient: authenticate success, global_id 54162 2024-09-15T04:00:54.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b88a80640 10 monclient: _renew_subs 2024-09-15T04:00:54.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b88a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:54.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b88a80640 1 -- 172.21.15.110:0/3021348388 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b84098d10 con 0x7f8b841369e0 2024-09-15T04:00:54.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b80d99640 4 set_mon_vals no callback set 2024-09-15T04:00:54.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:54.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b727fc640 10 monclient: dump: 2024-09-15T04:00:54.210 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.210 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.210 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.210 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.210 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.211 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.211 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.211 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.211 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.211 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b88a80640 10 monclient: _renew_subs 2024-09-15T04:00:54.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b88a80640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:54.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.192+0000 7f8b88a80640 1 -- 172.21.15.110:0/3021348388 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b840998e0 con 0x7f8b841369e0 2024-09-15T04:00:54.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b727fc640 1 -- 172.21.15.110:0/3021348388 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f8b7c013a30 con 0x7f8b841369e0 2024-09-15T04:00:54.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b88a80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:54.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b88a80640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84097c90 0x7f8b841346c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b88a80640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b88a80640 1 -- 172.21.15.110:0/3021348388 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8409a970 con 0x7f8b84097c90 2024-09-15T04:00:54.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b727fc640 1 --2- 172.21.15.110:0/3021348388 >> v2:172.21.15.177:6812/28201 conn(0x7f8b64038700 0x7f8b6403abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b8259c640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84097c90 0x7f8b841346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b727fc640 1 -- 172.21.15.110:0/3021348388 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f8b7c04ba90 con 0x7f8b841369e0 2024-09-15T04:00:54.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b8259c640 10 monclient: get_auth_request con 0x7f8b84097c90 auth_method 0 2024-09-15T04:00:54.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b8259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b8259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b8259c640 10 monclient(hunting): handle_auth_done global_id 4850 payload 824 2024-09-15T04:00:54.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b8259c640 1 -- 172.21.15.110:0/3021348388 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b84098060 con 0x7f8b84097c90 2024-09-15T04:00:54.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b81d9b640 1 --2- 172.21.15.110:0/3021348388 >> v2:172.21.15.177:6812/28201 conn(0x7f8b64038700 0x7f8b6403abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.193+0000 7f8b81d9b640 10 monclient: get_auth_request con 0x7f8b64038700 auth_method 0 2024-09-15T04:00:54.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.196+0000 7f8b8259c640 1 -- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84097c90 msgr2=0x7f8b841346c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.196+0000 7f8b8259c640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84097c90 0x7f8b841346c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400fa50 tx=0x7f8b74007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.196+0000 7f8b727fc640 1 -- 172.21.15.110:0/3021348388 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7400dda0 con 0x7f8b84097c90 2024-09-15T04:00:54.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.196+0000 7f8b727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.196+0000 7f8b727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:54.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.196+0000 7f8b727fc640 10 monclient: dump: 2024-09-15T04:00:54.213 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.213 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.213 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.213 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.213 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.213 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.214 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.214 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.214 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.214 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.196+0000 7f8b727fc640 1 -- 172.21.15.110:0/3021348388 <== mon.2 v2:172.21.15.110:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8b74014470 con 0x7f8b84097c90 2024-09-15T04:00:54.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.196+0000 7f8b727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:54.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.196+0000 7f8b727fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:00:54.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.196+0000 7f8b727fc640 1 -- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84097c90 msgr2=0x7f8b841346c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.196+0000 7f8b727fc640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84097c90 0x7f8b841346c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400fa50 tx=0x7f8b74007e10 comp rx=0 tx=0).stop 2024-09-15T04:00:54.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.196+0000 7f8b727fc640 1 -- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84097c90 msgr2=0x7f8b841346c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:00:54.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.196+0000 7f8b727fc640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84097c90 0x7f8b841346c0 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400fa50 tx=0x7f8b74007e10 comp rx=0 tx=0).stop 2024-09-15T04:00:54.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.196+0000 7f8b81d9b640 1 --2- 172.21.15.110:0/3021348388 >> v2:172.21.15.177:6812/28201 conn(0x7f8b64038700 0x7f8b6403abb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8b78034000 tx=0x7f8b78032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.202+0000 7f8b88a80640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:00:54.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.202+0000 7f8b88a80640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84134c00 0x7f8b84135000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.202+0000 7f8b88a80640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.202+0000 7f8b88a80640 1 -- 172.21.15.110:0/3021348388 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b84063e80 con 0x7f8b84134c00 2024-09-15T04:00:54.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.202+0000 7f8b8259c640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84134c00 0x7f8b84135000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.202+0000 7f8b8259c640 10 monclient: get_auth_request con 0x7f8b84134c00 auth_method 0 2024-09-15T04:00:54.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.202+0000 7f8b8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.202+0000 7f8b8259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.202+0000 7f8b8259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.202+0000 7f8b8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.202+0000 7f8b8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.202+0000 7f8b8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.202+0000 7f8b8259c640 10 monclient(hunting): handle_auth_done global_id 4853 payload 824 2024-09-15T04:00:54.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.202+0000 7f8b8259c640 1 -- 172.21.15.110:0/3021348388 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8b840647e0 con 0x7f8b84134c00 2024-09-15T04:00:54.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.204+0000 7f8b8259c640 1 -- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84134c00 msgr2=0x7f8b84135000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.204+0000 7f8b8259c640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84134c00 0x7f8b84135000 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8b74019000 tx=0x7f8b7401a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.204+0000 7f8b727fc640 1 -- 172.21.15.110:0/3021348388 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7401ed30 con 0x7f8b84134c00 2024-09-15T04:00:54.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.204+0000 7f8b727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.204+0000 7f8b727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:54.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.204+0000 7f8b727fc640 10 monclient: dump: 2024-09-15T04:00:54.218 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.218 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.218 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.218 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.218 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.218 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.218 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.219 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.219 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.219 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.204+0000 7f8b727fc640 1 -- 172.21.15.110:0/3021348388 <== mon.2 v2:172.21.15.110:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f8b74020410 con 0x7f8b84134c00 2024-09-15T04:00:54.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.204+0000 7f8b727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:00:54.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.204+0000 7f8b727fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:00:54.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.204+0000 7f8b727fc640 1 -- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84134c00 msgr2=0x7f8b84135000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.204+0000 7f8b727fc640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84134c00 0x7f8b84135000 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8b74019000 tx=0x7f8b7401a000 comp rx=0 tx=0).stop 2024-09-15T04:00:54.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.204+0000 7f8b727fc640 1 -- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84134c00 msgr2=0x7f8b84135000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:00:54.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.204+0000 7f8b727fc640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8b84134c00 0x7f8b84135000 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8b74019000 tx=0x7f8b7401a000 comp rx=0 tx=0).stop 2024-09-15T04:00:54.219 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:00:54.219 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:00:54.220 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:00:54.220 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T04:00:54.220 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 22, 2024-09-15T04:00:54.220 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:00:54.220 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:00:54.220 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:00:54.220 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:00:54.220 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.220 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 4, 2024-09-15T04:00:54.220 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:00:54.220 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:00:54.220 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:00:54.220 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:54.220 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:54.221 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:54.221 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:54.221 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:54.221 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:54.221 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:54.221 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:54.221 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:54.221 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.221 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:00:54.221 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:00:54.221 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:54.221 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:54.221 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:54.222 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:54.222 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:54.222 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:54.222 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:54.222 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:54.222 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:54.222 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.222 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.222 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:00:54.222 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:00:54.222 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:00:54.222 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:00:54.222 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:00:54.222 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:00:54.223 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:00:54.223 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:00:54.223 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:00:54.223 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:00:54.223 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:00:54.223 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:00:54.223 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:00:54.223 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:00:54.223 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:00:54.223 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:00:54.223 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:00:54.223 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:54.224 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:54.224 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:54.224 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:54.224 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:54.224 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:54.224 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:54.224 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:54.224 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:54.224 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.224 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:00:54.224 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.224 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:00:54.224 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.224 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:00:54.225 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:00:54.225 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:54.225 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:54.225 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.225 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:54.225 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:00:54.225 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.225 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.225 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.225 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:54.225 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:00:54.225 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.225 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.226 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.226 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:00:54.226 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:00:54.226 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:54.226 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:54.226 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:54.226 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.226 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.226 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:00:54.226 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:00:54.226 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:54.226 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:54.227 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.227 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:54.227 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:00:54.227 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.227 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.227 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:54.227 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:00:54.227 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.227 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.227 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.227 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.227 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:00:54.227 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:00:54.228 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:54.228 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:54.228 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:54.228 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.228 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:00:54.228 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:00:54.228 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:54.228 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:54.228 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.228 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:54.228 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:00:54.229 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.229 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.229 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.229 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:54.229 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:00:54.229 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.229 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.229 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.229 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.229 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:00:54.229 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:00:54.229 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T04:00:54.230 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.231 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:54.231 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:54.231 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:00:54.231 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.231 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.231 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:00:54.231 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.231 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:54.231 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:54.231 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 3 2024-09-15T04:00:54.231 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.231 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.232 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.232 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:00:54.232 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:00:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.205+0000 7f8b88a80640 1 -- 172.21.15.110:0/3021348388 >> v2:172.21.15.177:6812/28201 conn(0x7f8b64038700 msgr2=0x7f8b6403abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.205+0000 7f8b88a80640 1 --2- 172.21.15.110:0/3021348388 >> v2:172.21.15.177:6812/28201 conn(0x7f8b64038700 0x7f8b6403abb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8b78034000 tx=0x7f8b78032000 comp rx=0 tx=0).stop 2024-09-15T04:00:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.205+0000 7f8b88a80640 10 monclient: shutdown 2024-09-15T04:00:54.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.205+0000 7f8b88a80640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.205+0000 7f8b88a80640 1 -- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b841369e0 msgr2=0x7f8b84097750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.205+0000 7f8b88a80640 1 --2- 172.21.15.110:0/3021348388 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b841369e0 0x7f8b84097750 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c007c70 tx=0x7f8b7c009310 comp rx=0 tx=0).stop 2024-09-15T04:00:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.205+0000 7f8b82d9d640 1 -- 172.21.15.110:0/3021348388 reap_dead start 2024-09-15T04:00:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.205+0000 7f8b82d9d640 1 -- 172.21.15.110:0/3021348388 reap_dead start 2024-09-15T04:00:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.205+0000 7f8b88a80640 1 -- 172.21.15.110:0/3021348388 shutdown_connections 2024-09-15T04:00:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.205+0000 7f8b88a80640 1 -- 172.21.15.110:0/3021348388 >> 172.21.15.110:0/3021348388 conn(0x7f8b8406e860 msgr2=0x7f8b8409d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.205+0000 7f8b88a80640 1 -- 172.21.15.110:0/3021348388 shutdown_connections 2024-09-15T04:00:54.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.205+0000 7f8b88a80640 1 -- 172.21.15.110:0/3021348388 wait complete. 2024-09-15T04:00:54.234 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:00:54.234 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T04:00:54.235 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T04:00:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.328+0000 7f6e52e57640 10 monclient: get_monmap_and_config 2024-09-15T04:00:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.328+0000 7f6e52e57640 10 monclient: build_initial_monmap 2024-09-15T04:00:54.329 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.328+0000 7f6e52e57640 10 monclient: monmap: 2024-09-15T04:00:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:00:54.329115+0000 2024-09-15T04:00:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:00:54.329115+0000 2024-09-15T04:00:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:00:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:00:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:00:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:00:54.330 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:00:54.331 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.328+0000 7f6e52e57640 1 Processor -- start 2024-09-15T04:00:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.328+0000 7f6e52e57640 1 -- start start 2024-09-15T04:00:54.331 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.328+0000 7f6e52e57640 10 monclient: init 2024-09-15T04:00:54.332 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e52e57640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e52e57640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-15T04:00:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e52e57640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6e4c09eaa0 0x7f6e4c09ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e52e57640 10 monclient(hunting): picked mon.noname-a con 0x7f6e4c09eaa0 addr v2:172.21.15.110:3300/0 2024-09-15T04:00:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e52e57640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e4c06a080 addr v1:172.21.15.110:6789/0 2024-09-15T04:00:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e52e57640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c06ec00 0x7f6e4c13cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e52e57640 10 monclient(hunting): picked mon.noname-b con 0x7f6e4c06ec00 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e52e57640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4c09c6c0 con 0x7f6e4c06a080 2024-09-15T04:00:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e52e57640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e52e57640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e4c06f000 con 0x7f6e4c09eaa0 2024-09-15T04:00:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e52e57640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.333 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e52e57640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c09dce0 con 0x7f6e4c06ec00 2024-09-15T04:00:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e52e57640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e52e57640 10 monclient(hunting): authenticate will time out at 877.973414s 2024-09-15T04:00:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c06ec00 0x7f6e4c13cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e50bf3640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6e4c09eaa0 0x7f6e4c09ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c06ec00 0x7f6e4c13cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50818/0 (socket says 172.21.15.110:50818) 2024-09-15T04:00:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 1 -- 172.21.15.110:0/1604467909 learned_addr learned my addr 172.21.15.110:0/1604467909 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 10 monclient(hunting): get_auth_request con 0x7f6e4c06ec00 auth_method 0 2024-09-15T04:00:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.334 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e50bf3640 10 monclient(hunting): get_auth_request con 0x7f6e4c09eaa0 auth_method 0 2024-09-15T04:00:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e50bf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e50bf3640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e50bf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 10 monclient(hunting): handle_auth_done global_id 4856 payload 274 2024-09-15T04:00:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 1 -- 172.21.15.110:0/1604467909 >> v2:172.21.15.110:3300/0 conn(0x7f6e4c09eaa0 msgr2=0x7f6e4c09ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 1 --2- 172.21.15.110:0/1604467909 >> v2:172.21.15.110:3300/0 conn(0x7f6e4c09eaa0 0x7f6e4c09ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 1 -- 172.21.15.110:0/1604467909 >> v1:172.21.15.110:6789/0 conn(0x7f6e4c06a080 legacy=0x7f6e4c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:54.335 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 10 monclient: _finish_hunting 0 2024-09-15T04:00:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 1 monclient: found mon.noname-b 2024-09-15T04:00:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:00:54.336 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.329+0000 7f6e513f4640 1 -- 172.21.15.110:0/1604467909 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c09cad0 con 0x7f6e4c06ec00 2024-09-15T04:00:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e513f4640 1 -- 172.21.15.110:0/1604467909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c06ec00 msgr2=0x7f6e4c13cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e513f4640 1 --2- 172.21.15.110:0/1604467909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c06ec00 0x7f6e4c13cde0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6e44009910 tx=0x7f6e440091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=589f3c5319e74a7e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e437fe640 1 -- 172.21.15.110:0/1604467909 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4402aea0 con 0x7f6e4c06ec00 2024-09-15T04:00:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:00:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e437fe640 10 monclient: dump: 2024-09-15T04:00:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.343 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.344 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e437fe640 10 monclient: _finish_auth 0 2024-09-15T04:00:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e437fe640 1 -- 172.21.15.110:0/1604467909 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e44032440 con 0x7f6e4c06ec00 2024-09-15T04:00:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e52e57640 5 monclient: authenticate success, global_id 4856 2024-09-15T04:00:54.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e52e57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:54.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e437fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:54.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e52e57640 10 monclient: get_monmap_and_config success 2024-09-15T04:00:54.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e437fe640 1 -- 172.21.15.110:0/1604467909 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e440327e0 con 0x7f6e4c06ec00 2024-09-15T04:00:54.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e52e57640 4 set_mon_vals no callback set 2024-09-15T04:00:54.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:54.345 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.332+0000 7f6e437fe640 10 monclient: dump: 2024-09-15T04:00:54.345 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.345 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.345 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.345 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.345 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.346 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.333+0000 7f6e52e57640 10 monclient: shutdown 2024-09-15T04:00:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.333+0000 7f6e52e57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.333+0000 7f6e52e57640 1 -- 172.21.15.110:0/1604467909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c06ec00 msgr2=0x7f6e4c13cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.333+0000 7f6e52e57640 1 --2- 172.21.15.110:0/1604467909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c06ec00 0x7f6e4c13cde0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6e44009910 tx=0x7f6e440091d0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.333+0000 7f6e52e57640 1 -- 172.21.15.110:0/1604467909 shutdown_connections 2024-09-15T04:00:54.346 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.333+0000 7f6e52e57640 1 --2- 172.21.15.110:0/1604467909 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c06ec00 0x7f6e4c13cde0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.333+0000 7f6e52e57640 1 --2- 172.21.15.110:0/1604467909 >> v2:172.21.15.110:3300/0 conn(0x7f6e4c09eaa0 0x7f6e4c09ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.333+0000 7f6e52e57640 1 -- 172.21.15.110:0/1604467909 >> 172.21.15.110:0/1604467909 conn(0x7f6e4c0a4fb0 msgr2=0x7f6e4c0a53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.333+0000 7f6e52e57640 1 -- 172.21.15.110:0/1604467909 shutdown_connections 2024-09-15T04:00:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.333+0000 7f6e52e57640 1 -- 172.21.15.110:0/1604467909 wait complete. 2024-09-15T04:00:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.333+0000 7f6e52e57640 10 monclient: build_initial_monmap 2024-09-15T04:00:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.333+0000 7f6e52e57640 10 monclient: monmap: 2024-09-15T04:00:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:00:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:00:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:00:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.347 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:00:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:00:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:00:54.348 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 1 Processor -- start 2024-09-15T04:00:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 1 -- start start 2024-09-15T04:00:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 10 monclient: init 2024-09-15T04:00:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T04:00:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e4c06a080 0x7f6e4c13cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 10 monclient(hunting): picked mon.noname-a con 0x7f6e4c06a080 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:54.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c06ec00 0x7f6e4c09a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 10 monclient(hunting): picked mon.noname-b con 0x7f6e4c06ec00 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c09eaa0 0x7f6e4c09ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 10 monclient(hunting): picked mon.noname-c con 0x7f6e4c09eaa0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c06f160 con 0x7f6e4c06a080 2024-09-15T04:00:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c0a3d50 con 0x7f6e4c06ec00 2024-09-15T04:00:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c125be0 con 0x7f6e4c09eaa0 2024-09-15T04:00:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e52e57640 10 monclient(hunting): authenticate will time out at 877.978905s 2024-09-15T04:00:54.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e50bf3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e4c06a080 0x7f6e4c13cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e50bf3640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e4c06a080 0x7f6e4c13cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54060/0 (socket says 172.21.15.110:54060) 2024-09-15T04:00:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e50bf3640 1 -- 172.21.15.110:0/3300506046 learned_addr learned my addr 172.21.15.110:0/3300506046 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e50bf3640 10 monclient(hunting): get_auth_request con 0x7f6e4c06a080 auth_method 0 2024-09-15T04:00:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e50bf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e50bf3640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e50bf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.334+0000 7f6e513f4640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c09eaa0 0x7f6e4c09ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 10 monclient(hunting): get_auth_request con 0x7f6e4c09eaa0 auth_method 0 2024-09-15T04:00:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e50bf3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e50bf3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e50bf3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 10 monclient(hunting): handle_auth_done global_id 4859 payload 824 2024-09-15T04:00:54.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e43fff640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c06ec00 0x7f6e4c09a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 1 -- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c06ec00 msgr2=0x7f6e4c09a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c06ec00 0x7f6e4c09a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 1 -- 172.21.15.110:0/3300506046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e4c06a080 msgr2=0x7f6e4c13cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e4c06a080 0x7f6e4c13cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 10 monclient: _finish_hunting 0 2024-09-15T04:00:54.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 1 monclient: found mon.noname-c 2024-09-15T04:00:54.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:54.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:00:54.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e513f4640 1 -- 172.21.15.110:0/3300506046 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e44009450 con 0x7f6e4c09eaa0 2024-09-15T04:00:54.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e43fff640 10 monclient: get_auth_request con 0x7f6e4c06ec00 auth_method 0 2024-09-15T04:00:54.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e50bf3640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6e4c06a080 0x7f6e4c13cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:00:54.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.335+0000 7f6e43fff640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c06ec00 0x7f6e4c09a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:00:54.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e513f4640 1 -- 172.21.15.110:0/3300506046 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c09eaa0 msgr2=0x7f6e4c09ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e513f4640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c09eaa0 0x7f6e4c09ab80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6e44031000 tx=0x7f6e44032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e40ff9640 1 -- 172.21.15.110:0/3300506046 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4402aea0 con 0x7f6e4c09eaa0 2024-09-15T04:00:54.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e40ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:00:54.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e40ff9640 10 monclient: dump: 2024-09-15T04:00:54.353 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.353 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.353 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.353 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.353 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.353 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.353 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.354 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.354 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.354 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e40ff9640 10 monclient: _finish_auth 0 2024-09-15T04:00:54.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:54.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e40ff9640 1 -- 172.21.15.110:0/3300506046 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e44031e90 con 0x7f6e4c09eaa0 2024-09-15T04:00:54.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e52e57640 5 monclient: authenticate success, global_id 4859 2024-09-15T04:00:54.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e52e57640 10 monclient: _renew_subs 2024-09-15T04:00:54.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e52e57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:54.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e52e57640 1 -- 172.21.15.110:0/3300506046 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4c126110 con 0x7f6e4c09eaa0 2024-09-15T04:00:54.354 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e40ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e40ff9640 1 -- 172.21.15.110:0/3300506046 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4403d650 con 0x7f6e4c09eaa0 2024-09-15T04:00:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e40ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:00:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e40ff9640 10 monclient: dump: 2024-09-15T04:00:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.355 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.356 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e42ffd640 4 set_mon_vals no callback set 2024-09-15T04:00:54.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e52e57640 10 monclient: _renew_subs 2024-09-15T04:00:54.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e52e57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:00:54.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.337+0000 7f6e52e57640 1 -- 172.21.15.110:0/3300506046 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4c13c0d0 con 0x7f6e4c09eaa0 2024-09-15T04:00:54.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.338+0000 7f6e40ff9640 1 -- 172.21.15.110:0/3300506046 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f6e44039040 con 0x7f6e4c09eaa0 2024-09-15T04:00:54.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.338+0000 7f6e52e57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:54.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.338+0000 7f6e52e57640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c124c60 0x7f6e4c02aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.338+0000 7f6e52e57640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.338+0000 7f6e52e57640 1 -- 172.21.15.110:0/3300506046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c124160 con 0x7f6e4c124c60 2024-09-15T04:00:54.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.338+0000 7f6e40ff9640 1 --2- 172.21.15.110:0/3300506046 >> v2:172.21.15.177:6812/28201 conn(0x7f6e38038700 0x7f6e3803abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.338+0000 7f6e40ff9640 1 -- 172.21.15.110:0/3300506046 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f6e4406b660 con 0x7f6e4c09eaa0 2024-09-15T04:00:54.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.338+0000 7f6e50bf3640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c124c60 0x7f6e4c02aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.338+0000 7f6e50bf3640 10 monclient: get_auth_request con 0x7f6e4c124c60 auth_method 0 2024-09-15T04:00:54.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.338+0000 7f6e50bf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.338+0000 7f6e50bf3640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.338+0000 7f6e50bf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.338+0000 7f6e43fff640 1 --2- 172.21.15.110:0/3300506046 >> v2:172.21.15.177:6812/28201 conn(0x7f6e38038700 0x7f6e3803abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.339+0000 7f6e43fff640 10 monclient: get_auth_request con 0x7f6e38038700 auth_method 0 2024-09-15T04:00:54.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.339+0000 7f6e50bf3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.339+0000 7f6e50bf3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.339+0000 7f6e50bf3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.339+0000 7f6e50bf3640 10 monclient(hunting): handle_auth_done global_id 4822 payload 824 2024-09-15T04:00:54.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.339+0000 7f6e50bf3640 1 -- 172.21.15.110:0/3300506046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e4c0647e0 con 0x7f6e4c124c60 2024-09-15T04:00:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.341+0000 7f6e43fff640 1 --2- 172.21.15.110:0/3300506046 >> v2:172.21.15.177:6812/28201 conn(0x7f6e38038700 0x7f6e3803abb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c0027e0 tx=0x7f6e3c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.341+0000 7f6e50bf3640 1 -- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c124c60 msgr2=0x7f6e4c02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.341+0000 7f6e50bf3640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c124c60 0x7f6e4c02aa40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6e480027b0 tx=0x7f6e48009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.341+0000 7f6e40ff9640 1 -- 172.21.15.110:0/3300506046 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e48002e80 con 0x7f6e4c124c60 2024-09-15T04:00:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.341+0000 7f6e40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.341+0000 7f6e40ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:00:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.341+0000 7f6e40ff9640 10 monclient: dump: 2024-09-15T04:00:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.358 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.359 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.359 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.359 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.359 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.342+0000 7f6e40ff9640 1 -- 172.21.15.110:0/3300506046 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f6e48010ce0 con 0x7f6e4c124c60 2024-09-15T04:00:54.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.342+0000 7f6e40ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:54.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.342+0000 7f6e40ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:00:54.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.342+0000 7f6e40ff9640 1 -- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c124c60 msgr2=0x7f6e4c02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.342+0000 7f6e40ff9640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c124c60 0x7f6e4c02aa40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6e480027b0 tx=0x7f6e48009920 comp rx=0 tx=0).stop 2024-09-15T04:00:54.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.342+0000 7f6e40ff9640 1 -- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c124c60 msgr2=0x7f6e4c02aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:00:54.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.342+0000 7f6e40ff9640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c124c60 0x7f6e4c02aa40 secure :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6e480027b0 tx=0x7f6e48009920 comp rx=0 tx=0).stop 2024-09-15T04:00:54.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.347+0000 7f6e52e57640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:00:54.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.347+0000 7f6e52e57640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c09b0c0 0x7f6e4c05e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.347+0000 7f6e52e57640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.347+0000 7f6e52e57640 1 -- 172.21.15.110:0/3300506046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4c063e80 con 0x7f6e4c09b0c0 2024-09-15T04:00:54.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.348+0000 7f6e50bf3640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c09b0c0 0x7f6e4c05e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.348+0000 7f6e50bf3640 10 monclient: get_auth_request con 0x7f6e4c09b0c0 auth_method 0 2024-09-15T04:00:54.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.348+0000 7f6e50bf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.348+0000 7f6e50bf3640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.348+0000 7f6e50bf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.348+0000 7f6e50bf3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.348+0000 7f6e50bf3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.348+0000 7f6e50bf3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.348+0000 7f6e50bf3640 10 monclient(hunting): handle_auth_done global_id 4825 payload 824 2024-09-15T04:00:54.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.348+0000 7f6e50bf3640 1 -- 172.21.15.110:0/3300506046 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6e4c09b4c0 con 0x7f6e4c09b0c0 2024-09-15T04:00:54.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.351+0000 7f6e50bf3640 1 -- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c09b0c0 msgr2=0x7f6e4c05e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.351+0000 7f6e50bf3640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c09b0c0 0x7f6e4c05e310 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6e480027b0 tx=0x7f6e4801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.351+0000 7f6e40ff9640 1 -- 172.21.15.110:0/3300506046 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e48015bb0 con 0x7f6e4c09b0c0 2024-09-15T04:00:54.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.351+0000 7f6e40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.351+0000 7f6e40ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:00:54.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.351+0000 7f6e40ff9640 10 monclient: dump: 2024-09-15T04:00:54.364 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.365 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.365 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.365 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.365 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.365 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.365 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.365 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.365 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.365 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.351+0000 7f6e40ff9640 1 -- 172.21.15.110:0/3300506046 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f6e48015d10 con 0x7f6e4c09b0c0 2024-09-15T04:00:54.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.351+0000 7f6e40ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:00:54.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.351+0000 7f6e40ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:00:54.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.351+0000 7f6e40ff9640 1 -- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c09b0c0 msgr2=0x7f6e4c05e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.351+0000 7f6e40ff9640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c09b0c0 0x7f6e4c05e310 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6e480027b0 tx=0x7f6e4801b000 comp rx=0 tx=0).stop 2024-09-15T04:00:54.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.351+0000 7f6e40ff9640 1 -- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c09b0c0 msgr2=0x7f6e4c05e310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:00:54.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.351+0000 7f6e40ff9640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6e4c09b0c0 0x7f6e4c05e310 secure :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6e480027b0 tx=0x7f6e4801b000 comp rx=0 tx=0).stop 2024-09-15T04:00:54.368 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:00:54.368 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:00:54.369 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:00:54.369 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T04:00:54.369 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 22, 2024-09-15T04:00:54.369 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:00:54.369 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:00:54.369 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:00:54.369 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:00:54.369 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.369 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 4, 2024-09-15T04:00:54.369 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:00:54.369 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:00:54.369 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:00:54.370 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:54.370 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:54.370 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:54.370 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:54.370 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:54.370 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:54.370 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:54.370 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:54.370 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:54.370 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.370 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:00:54.370 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:00:54.370 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:54.370 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:54.371 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:54.371 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:54.371 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:54.371 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:54.371 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:54.371 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:54.371 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:54.371 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.371 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.371 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:00:54.371 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:00:54.371 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:00:54.371 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:00:54.371 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:00:54.372 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:00:54.372 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:00:54.372 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:00:54.372 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:00:54.372 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:00:54.372 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:00:54.372 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:00:54.372 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:00:54.372 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:00:54.372 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:00:54.372 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:00:54.372 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:00:54.372 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:00:54.373 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:00:54.373 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:00:54.373 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:00:54.373 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:00:54.373 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:00:54.373 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:00:54.373 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:00:54.373 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:00:54.373 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.373 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:00:54.373 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.373 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:00:54.373 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.373 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:00:54.374 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:00:54.374 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:54.374 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:54.374 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.374 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:54.374 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:00:54.374 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.374 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.374 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.375 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:54.375 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:00:54.375 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.375 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.375 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.375 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.375 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:00:54.375 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:00:54.375 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:54.375 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:54.375 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:54.375 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.375 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.376 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:00:54.376 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:00:54.376 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:54.376 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:54.376 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.376 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:54.376 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:00:54.376 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.376 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.376 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.376 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:54.376 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:00:54.376 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.376 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.377 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.377 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.377 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:00:54.377 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:00:54.377 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:54.377 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:54.377 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:54.377 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.377 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.377 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:00:54.377 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:00:54.377 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:00:54.377 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:00:54.377 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:00:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:00:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:00:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:00:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:00:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:00:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:00:54.378 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T04:00:54.379 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.380 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:54.380 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:54.380 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 5 2024-09-15T04:00:54.380 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.380 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.380 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:00:54.380 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.380 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:54.380 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:54.380 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 5 2024-09-15T04:00:54.380 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.380 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:00:54.380 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": [ 2024-09-15T04:00:54.381 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:00:54.381 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:00:54.381 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:00:54.381 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:00:54.381 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:00:54.381 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:00:54.381 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:00:54.381 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:00:54.381 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:00:54.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.352+0000 7f6e52e57640 1 -- 172.21.15.110:0/3300506046 >> v2:172.21.15.177:6812/28201 conn(0x7f6e38038700 msgr2=0x7f6e3803abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.352+0000 7f6e52e57640 1 --2- 172.21.15.110:0/3300506046 >> v2:172.21.15.177:6812/28201 conn(0x7f6e38038700 0x7f6e3803abb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c0027e0 tx=0x7f6e3c009040 comp rx=0 tx=0).stop 2024-09-15T04:00:54.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.352+0000 7f6e52e57640 10 monclient: shutdown 2024-09-15T04:00:54.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.352+0000 7f6e52e57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:54.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.352+0000 7f6e52e57640 1 -- 172.21.15.110:0/3300506046 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c09eaa0 msgr2=0x7f6e4c09ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.352+0000 7f6e52e57640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c09eaa0 0x7f6e4c09ab80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6e44031000 tx=0x7f6e44032000 comp rx=0 tx=0).stop 2024-09-15T04:00:54.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.352+0000 7f6e513f4640 1 -- 172.21.15.110:0/3300506046 reap_dead start 2024-09-15T04:00:54.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.352+0000 7f6e52e57640 1 -- 172.21.15.110:0/3300506046 shutdown_connections 2024-09-15T04:00:54.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.352+0000 7f6e52e57640 1 --2- 172.21.15.110:0/3300506046 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6e4c09eaa0 0x7f6e4c09ab80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.352+0000 7f6e52e57640 1 -- 172.21.15.110:0/3300506046 >> 172.21.15.110:0/3300506046 conn(0x7f6e4c0a4fb0 msgr2=0x7f6e4c06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:54.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.352+0000 7f6e52e57640 1 -- 172.21.15.110:0/3300506046 shutdown_connections 2024-09-15T04:00:54.383 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.352+0000 7f6e52e57640 1 -- 172.21.15.110:0/3300506046 wait complete. 2024-09-15T04:00:54.384 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:00:54.384 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T04:00:54.384 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T04:00:54.384 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T04:00:54.384 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T04:00:54.384 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T04:00:54.385 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T04:00:54.395 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T04:00:54.395 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T04:00:54.396 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:00:54.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.526+0000 7fcd90a60640 10 monclient: get_monmap_and_config 2024-09-15T04:00:54.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.526+0000 7fcd90a60640 10 monclient: build_initial_monmap 2024-09-15T04:00:54.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.526+0000 7fcd90a60640 10 monclient: monmap: 2024-09-15T04:00:54.528 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:54.528 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.528 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:00:54.527794+0000 2024-09-15T04:00:54.528 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:00:54.527794+0000 2024-09-15T04:00:54.528 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:54.528 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.528 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:00:54.528 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:00:54.528 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:00:54.528 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:00:54.528 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:00:54.528 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.527+0000 7fcd90a60640 1 Processor -- start 2024-09-15T04:00:54.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.527+0000 7fcd90a60640 1 -- start start 2024-09-15T04:00:54.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.527+0000 7fcd90a60640 10 monclient: init 2024-09-15T04:00:54.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.527+0000 7fcd90a60640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:54.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.527+0000 7fcd90a60640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-15T04:00:54.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.527+0000 7fcd90a60640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c09bf10 0x7fcd8c09c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.528+0000 7fcd90a60640 10 monclient(hunting): picked mon.noname-b con 0x7fcd8c09bf10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:54.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.528+0000 7fcd90a60640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcd8c06ed10 addr v1:172.21.15.110:6789/0 2024-09-15T04:00:54.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.528+0000 7fcd90a60640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcd8c06d050 addr v1:172.21.15.177:6789/0 2024-09-15T04:00:54.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.528+0000 7fcd90a60640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8c06c0a0 con 0x7fcd8c06ed10 2024-09-15T04:00:54.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.528+0000 7fcd90a60640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8c06a8f0 con 0x7fcd8c06d050 2024-09-15T04:00:54.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.528+0000 7fcd90a60640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.528+0000 7fcd90a60640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c06ae00 con 0x7fcd8c09bf10 2024-09-15T04:00:54.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.528+0000 7fcd90a60640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:54.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.528+0000 7fcd90a60640 10 monclient(hunting): authenticate will time out at 878.172075s 2024-09-15T04:00:54.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.528+0000 7fcd8a59c640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c09bf10 msgr2=0x7fcd8c09c2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:00:54.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.528+0000 7fcd8a59c640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c09bf10 0x7fcd8c09c2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:00:54.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.528+0000 7fcd89d9b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fcd8c06ed10 0x7fcd8c09c820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51226/0 (socket says 172.21.15.110:51226) 2024-09-15T04:00:54.531 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.528+0000 7fcd89d9b640 1 -- 172.21.15.110:0/4204961389 learned_addr learned my addr 172.21.15.110:0/4204961389 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:00:54.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.530+0000 7fcd89d9b640 1 -- 172.21.15.110:0/4204961389 >> v1:172.21.15.110:6789/0 conn(0x7fcd8c06ed10 legacy=0x7fcd8c09c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.530+0000 7fcd8ad9d640 1 -- 172.21.15.110:0/4204961389 >> v1:172.21.15.177:6789/0 conn(0x7fcd8c06d050 legacy=0x7fcd8c135270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.530+0000 7fcd8959a640 1 -- 172.21.15.110:0/4204961389 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64721986 0 0) 0x7fcd8c06c0a0 con 0x7fcd8c06ed10 2024-09-15T04:00:54.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.530+0000 7fcd8959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.530+0000 7fcd8959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.530+0000 7fcd8959a640 10 monclient(hunting): my global_id is 54171 2024-09-15T04:00:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.530+0000 7fcd8959a640 1 -- 172.21.15.110:0/4204961389 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd740034e0 con 0x7fcd8c06ed10 2024-09-15T04:00:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.530+0000 7fcd8959a640 1 -- 172.21.15.110:0/4204961389 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2510427064 0 0) 0x7fcd740034e0 con 0x7fcd8c06ed10 2024-09-15T04:00:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 1 -- 172.21.15.110:0/4204961389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c09bf10 msgr2=0x7fcd8c09c2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:00:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 1 --2- 172.21.15.110:0/4204961389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c09bf10 0x7fcd8c09c2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 1 -- 172.21.15.110:0/4204961389 >> v1:172.21.15.177:6789/0 conn(0x7fcd8c06d050 legacy=0x7fcd8c135270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient: _finish_hunting 0 2024-09-15T04:00:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 1 monclient: found mon.noname-a-legacy 2024-09-15T04:00:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.110:6789/0 2024-09-15T04:00:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 1 -- 172.21.15.110:0/4204961389 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8c06bdc0 con 0x7fcd8c06ed10 2024-09-15T04:00:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient: _finish_auth 0 2024-09-15T04:00:54.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 1 -- 172.21.15.110:0/4204961389 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3165669920 0 0) 0x7fcd80001720 con 0x7fcd8c06ed10 2024-09-15T04:00:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd90a60640 5 monclient: authenticate success, global_id 54171 2024-09-15T04:00:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd90a60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T04:00:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient: dump: 2024-09-15T04:00:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.540 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.541 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] but i'm connected to v1:172.21.15.110:6789/0, reconnecting 2024-09-15T04:00:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 1 -- 172.21.15.110:0/4204961389 >> v1:172.21.15.110:6789/0 conn(0x7fcd8c06ed10 legacy=0x7fcd8c09c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T04:00:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 1 --2- 172.21.15.110:0/4204961389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd74003a80 0x7fcd74003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient(hunting): picked mon.a con 0x7fcd74003a80 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 1 --2- 172.21.15.110:0/4204961389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd74004780 0x7fcd74004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient(hunting): picked mon.b con 0x7fcd74004780 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 1 --2- 172.21.15.110:0/4204961389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c09bf10 0x7fcd7400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient(hunting): picked mon.c con 0x7fcd8c09bf10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:54.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 1 -- 172.21.15.110:0/4204961389 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c06ae00 con 0x7fcd74003a80 2024-09-15T04:00:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 1 -- 172.21.15.110:0/4204961389 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd74005630 con 0x7fcd74004780 2024-09-15T04:00:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 1 -- 172.21.15.110:0/4204961389 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7400bf90 con 0x7fcd8c09bf10 2024-09-15T04:00:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8959a640 10 monclient(hunting): _finish_auth 0 2024-09-15T04:00:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd90a60640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8ad9d640 1 -- 172.21.15.110:0/4204961389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c09bf10 msgr2=0x7fcd7400b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:00:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8ad9d640 1 --2- 172.21.15.110:0/4204961389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c09bf10 0x7fcd7400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:00:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 1 --2- 172.21.15.110:0/4204961389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd74003a80 0x7fcd74003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 10 monclient(hunting): get_auth_request con 0x7fcd74003a80 auth_method 0 2024-09-15T04:00:54.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd89d9b640 1 --2- 172.21.15.110:0/4204961389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd74004780 0x7fcd74004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:00:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd89d9b640 10 monclient(hunting): get_auth_request con 0x7fcd74004780 auth_method 0 2024-09-15T04:00:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd89d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd89d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:00:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd89d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd89d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd89d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:00:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 10 monclient(hunting): handle_auth_done global_id 54171 payload 293 2024-09-15T04:00:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 1 -- 172.21.15.110:0/4204961389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c09bf10 msgr2=0x7fcd7400b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:00:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 1 --2- 172.21.15.110:0/4204961389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c09bf10 0x7fcd7400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 1 -- 172.21.15.110:0/4204961389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd74004780 msgr2=0x7fcd74004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:00:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 1 --2- 172.21.15.110:0/4204961389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd74004780 0x7fcd74004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 10 monclient: _finish_hunting 0 2024-09-15T04:00:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 1 monclient: found mon.a 2024-09-15T04:00:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 1 -- 172.21.15.110:0/4204961389 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd740034e0 con 0x7fcd74003a80 2024-09-15T04:00:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 10 monclient: _finish_auth 0 2024-09-15T04:00:54.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.531+0000 7fcd8a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd8a59c640 1 -- 172.21.15.110:0/4204961389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd74003a80 msgr2=0x7fcd74003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd8a59c640 1 --2- 172.21.15.110:0/4204961389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd74003a80 0x7fcd74003e90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0099c0 tx=0x7fcd7c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a63825c0c7a25e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd8959a640 1 -- 172.21.15.110:0/4204961389 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c003b80 con 0x7fcd74003a80 2024-09-15T04:00:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd8959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd8959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd8959a640 10 monclient: dump: 2024-09-15T04:00:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.545 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.546 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd90a60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd8959a640 1 -- 172.21.15.110:0/4204961389 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7c003d20 con 0x7fcd74003a80 2024-09-15T04:00:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd8959a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd90a60640 10 monclient: get_monmap_and_config success 2024-09-15T04:00:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd90a60640 4 set_mon_vals no callback set 2024-09-15T04:00:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd8959a640 1 -- 172.21.15.110:0/4204961389 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c02ab80 con 0x7fcd74003a80 2024-09-15T04:00:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd8959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd8959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd8959a640 10 monclient: dump: 2024-09-15T04:00:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.546 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.547 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd90a60640 10 monclient: shutdown 2024-09-15T04:00:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd90a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd90a60640 1 -- 172.21.15.110:0/4204961389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd74003a80 msgr2=0x7fcd74003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd90a60640 1 --2- 172.21.15.110:0/4204961389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd74003a80 0x7fcd74003e90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c0099c0 tx=0x7fcd7c009060 comp rx=0 tx=0).stop 2024-09-15T04:00:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.534+0000 7fcd8ad9d640 1 -- 172.21.15.110:0/4204961389 reap_dead start 2024-09-15T04:00:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.535+0000 7fcd90a60640 1 -- 172.21.15.110:0/4204961389 shutdown_connections 2024-09-15T04:00:54.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.535+0000 7fcd90a60640 1 -- 172.21.15.110:0/4204961389 >> 172.21.15.110:0/4204961389 conn(0x7fcd8c09d8f0 msgr2=0x7fcd8c09dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.535+0000 7fcd90a60640 1 -- 172.21.15.110:0/4204961389 shutdown_connections 2024-09-15T04:00:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.535+0000 7fcd90a60640 1 -- 172.21.15.110:0/4204961389 wait complete. 2024-09-15T04:00:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.535+0000 7fcd90a60640 10 monclient: build_initial_monmap 2024-09-15T04:00:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.535+0000 7fcd90a60640 10 monclient: monmap: 2024-09-15T04:00:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:00:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:00:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:00:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:00:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:00:54.548 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:00:54.549 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.535+0000 7fcd90a60640 1 Processor -- start 2024-09-15T04:00:54.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.535+0000 7fcd90a60640 1 -- start start 2024-09-15T04:00:54.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.535+0000 7fcd90a60640 10 monclient: init 2024-09-15T04:00:54.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:54.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T04:00:54.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd74003a80 0x7fcd8c13ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 10 monclient(hunting): picked mon.noname-b con 0x7fcd74003a80 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:54.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c12d550 0x7fcd8c12d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 10 monclient(hunting): picked mon.noname-c con 0x7fcd8c12d550 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:54.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd8c1170c0 0x7fcd8c117560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 10 monclient(hunting): picked mon.noname-a con 0x7fcd8c1170c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:54.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c13fd30 con 0x7fcd8c1170c0 2024-09-15T04:00:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c119580 con 0x7fcd74003a80 2024-09-15T04:00:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c11a950 con 0x7fcd8c12d550 2024-09-15T04:00:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd90a60640 10 monclient(hunting): authenticate will time out at 878.180270s 2024-09-15T04:00:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd89d9b640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c12d550 msgr2=0x7fcd8c12d920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:00:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd89d9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c12d550 0x7fcd8c12d920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:00:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd8c1170c0 0x7fcd8c117560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd8c1170c0 0x7fcd8c117560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54074/0 (socket says 172.21.15.110:54074) 2024-09-15T04:00:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 1 -- 172.21.15.110:0/3954292436 learned_addr learned my addr 172.21.15.110:0/3954292436 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:54.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 10 monclient(hunting): get_auth_request con 0x7fcd8c1170c0 auth_method 0 2024-09-15T04:00:54.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:54.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:54.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:54.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:54.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:54.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:54.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8a59c640 1 --2- 172.21.15.110:0/3954292436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd74003a80 0x7fcd8c13ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 10 monclient(hunting): handle_auth_done global_id 54174 payload 824 2024-09-15T04:00:54.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 1 -- 172.21.15.110:0/3954292436 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c12d550 msgr2=0x7fcd8c12d920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:00:54.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 1 --2- 172.21.15.110:0/3954292436 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c12d550 0x7fcd8c12d920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 1 -- 172.21.15.110:0/3954292436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd74003a80 msgr2=0x7fcd8c13ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 1 --2- 172.21.15.110:0/3954292436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd74003a80 0x7fcd8c13ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 10 monclient: _finish_hunting 0 2024-09-15T04:00:54.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 1 monclient: found mon.noname-a 2024-09-15T04:00:54.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:54.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:00:54.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8ad9d640 1 -- 172.21.15.110:0/3954292436 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8c099e50 con 0x7fcd8c1170c0 2024-09-15T04:00:54.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8a59c640 10 monclient: get_auth_request con 0x7fcd74003a80 auth_method 0 2024-09-15T04:00:54.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.536+0000 7fcd8a59c640 1 --2- 172.21.15.110:0/3954292436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd74003a80 0x7fcd8c13ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:00:54.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.538+0000 7fcd8ad9d640 1 -- 172.21.15.110:0/3954292436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd8c1170c0 msgr2=0x7fcd8c117560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:54.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.538+0000 7fcd8ad9d640 1 --2- 172.21.15.110:0/3954292436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd8c1170c0 0x7fcd8c117560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c06d490 tx=0x7fcd84007e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd7a7fc640 1 -- 172.21.15.110:0/3954292436 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8400cb00 con 0x7fcd8c1170c0 2024-09-15T04:00:54.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd7a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:00:54.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd7a7fc640 10 monclient: dump: 2024-09-15T04:00:54.555 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.555 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.555 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.555 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.555 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.555 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.555 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.555 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.555 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.556 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd7a7fc640 10 monclient: _finish_auth 0 2024-09-15T04:00:54.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:54.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd90a60640 5 monclient: authenticate success, global_id 54174 2024-09-15T04:00:54.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd7a7fc640 1 -- 172.21.15.110:0/3954292436 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8400cc80 con 0x7fcd8c1170c0 2024-09-15T04:00:54.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd90a60640 10 monclient: _renew_subs 2024-09-15T04:00:54.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd90a60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:54.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd90a60640 1 -- 172.21.15.110:0/3954292436 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd8c12de60 con 0x7fcd8c1170c0 2024-09-15T04:00:54.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:54.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd7a7fc640 1 -- 172.21.15.110:0/3954292436 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84007110 con 0x7fcd8c1170c0 2024-09-15T04:00:54.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:54.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd7a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:54.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd7a7fc640 10 monclient: dump: 2024-09-15T04:00:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:54.557 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd88d99640 4 set_mon_vals no callback set 2024-09-15T04:00:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd90a60640 10 monclient: _renew_subs 2024-09-15T04:00:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd90a60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:54.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd90a60640 1 -- 172.21.15.110:0/3954292436 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8c11a670 con 0x7fcd8c1170c0 2024-09-15T04:00:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.539+0000 7fcd7a7fc640 1 -- 172.21.15.110:0/3954292436 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fcd84007540 con 0x7fcd8c1170c0 2024-09-15T04:00:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.540+0000 7fcd7a7fc640 1 --2- 172.21.15.110:0/3954292436 >> v2:172.21.15.177:6812/28201 conn(0x7fcd6c0386b0 0x7fcd6c03ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.540+0000 7fcd90a60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.540+0000 7fcd90a60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.540+0000 7fcd90a60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.540+0000 7fcd90a60640 1 -- 172.21.15.110:0/3954292436 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd8c070950 con 0x7fcd8c1170c0 2024-09-15T04:00:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.540+0000 7fcd7a7fc640 1 -- 172.21.15.110:0/3954292436 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fcd8404a710 con 0x7fcd8c1170c0 2024-09-15T04:00:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.540+0000 7fcd8a59c640 1 --2- 172.21.15.110:0/3954292436 >> v2:172.21.15.177:6812/28201 conn(0x7fcd6c0386b0 0x7fcd6c03ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.540+0000 7fcd8a59c640 10 monclient: get_auth_request con 0x7fcd6c0386b0 auth_method 0 2024-09-15T04:00:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.540+0000 7fcd8a59c640 1 -- 172.21.15.110:0/3954292436 >> v2:172.21.15.177:6812/28201 conn(0x7fcd6c0386b0 msgr2=0x7fcd6c03ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T04:00:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.540+0000 7fcd8a59c640 1 -- 172.21.15.110:0/3954292436 >> v2:172.21.15.177:6812/28201 conn(0x7fcd6c0386b0 msgr2=0x7fcd6c03ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T04:00:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.540+0000 7fcd8a59c640 1 --2- 172.21.15.110:0/3954292436 >> v2:172.21.15.177:6812/28201 conn(0x7fcd6c0386b0 0x7fcd6c03ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:00:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.540+0000 7fcd8a59c640 1 --2- 172.21.15.110:0/3954292436 >> v2:172.21.15.177:6812/28201 conn(0x7fcd6c0386b0 0x7fcd6c03ab60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:00:54.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.544+0000 7fcd7a7fc640 1 -- 172.21.15.110:0/3954292436 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcd840077f0 con 0x7fcd8c1170c0 2024-09-15T04:00:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.544+0000 7fcd7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:54.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.544+0000 7fcd7a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:00:54.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.741+0000 7fcd8a59c640 1 --2- 172.21.15.110:0/3954292436 >> v2:172.21.15.177:6812/28201 conn(0x7fcd6c0386b0 0x7fcd6c03ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:54.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.741+0000 7fcd8a59c640 10 monclient: get_auth_request con 0x7fcd6c0386b0 auth_method 0 2024-09-15T04:00:54.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.743+0000 7fcd8a59c640 1 --2- 172.21.15.110:0/3954292436 >> v2:172.21.15.177:6812/28201 conn(0x7fcd6c0386b0 0x7fcd6c03ab60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c003940 tx=0x7fcd7c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.760+0000 7fcd90a60640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:00:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.760+0000 7fcd90a60640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:00:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.760+0000 7fcd90a60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.760+0000 7fcd90a60640 1 -- 172.21.15.110:0/3954292436 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcd8c06a6d0 con 0x7fcd8c1170c0 2024-09-15T04:00:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.760+0000 7fcd7a7fc640 1 -- 172.21.15.110:0/3954292436 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fcd840102c0 con 0x7fcd8c1170c0 2024-09-15T04:00:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.760+0000 7fcd7a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:00:54.761 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.760+0000 7fcd7a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:00:54.762 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:00:54.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.762+0000 7fcd90a60640 1 -- 172.21.15.110:0/3954292436 >> v2:172.21.15.177:6812/28201 conn(0x7fcd6c0386b0 msgr2=0x7fcd6c03ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.762+0000 7fcd90a60640 1 --2- 172.21.15.110:0/3954292436 >> v2:172.21.15.177:6812/28201 conn(0x7fcd6c0386b0 0x7fcd6c03ab60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c003940 tx=0x7fcd7c030000 comp rx=0 tx=0).stop 2024-09-15T04:00:54.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.762+0000 7fcd90a60640 10 monclient: shutdown 2024-09-15T04:00:54.763 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.762+0000 7fcd90a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:54.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.762+0000 7fcd90a60640 1 -- 172.21.15.110:0/3954292436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd8c1170c0 msgr2=0x7fcd8c117560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:54.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.762+0000 7fcd90a60640 1 --2- 172.21.15.110:0/3954292436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd8c1170c0 0x7fcd8c117560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c06d490 tx=0x7fcd84007e90 comp rx=0 tx=0).stop 2024-09-15T04:00:54.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.762+0000 7fcd90a60640 1 -- 172.21.15.110:0/3954292436 shutdown_connections 2024-09-15T04:00:54.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.762+0000 7fcd90a60640 1 --2- 172.21.15.110:0/3954292436 >> v2:172.21.15.177:6812/28201 conn(0x7fcd6c0386b0 0x7fcd6c03ab60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.762+0000 7fcd90a60640 1 --2- 172.21.15.110:0/3954292436 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd8c1170c0 0x7fcd8c117560 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.762+0000 7fcd90a60640 1 --2- 172.21.15.110:0/3954292436 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd8c12d550 0x7fcd8c12d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.762+0000 7fcd90a60640 1 --2- 172.21.15.110:0/3954292436 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd74003a80 0x7fcd8c13ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:54.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.762+0000 7fcd90a60640 1 -- 172.21.15.110:0/3954292436 >> 172.21.15.110:0/3954292436 conn(0x7fcd8c09d8f0 msgr2=0x7fcd8c09b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:54.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.762+0000 7fcd90a60640 1 -- 172.21.15.110:0/3954292436 shutdown_connections 2024-09-15T04:00:54.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:54.762+0000 7fcd90a60640 1 -- 172.21.15.110:0/3954292436 wait complete. 2024-09-15T04:00:57.778 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:00:57.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.880+0000 7f6889aaf640 10 monclient: get_monmap_and_config 2024-09-15T04:00:57.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.880+0000 7f6889aaf640 10 monclient: build_initial_monmap 2024-09-15T04:00:57.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.880+0000 7f6889aaf640 10 monclient: monmap: 2024-09-15T04:00:57.881 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:57.882 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:57.882 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:00:57.881607+0000 2024-09-15T04:00:57.882 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:00:57.881607+0000 2024-09-15T04:00:57.882 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:57.882 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:57.882 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:00:57.882 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:00:57.882 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:00:57.882 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:00:57.882 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:00:57.882 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:57.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.881+0000 7f6889aaf640 1 Processor -- start 2024-09-15T04:00:57.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.881+0000 7f6889aaf640 1 -- start start 2024-09-15T04:00:57.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.881+0000 7f6889aaf640 10 monclient: init 2024-09-15T04:00:57.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6889aaf640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:57.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6889aaf640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-15T04:00:57.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6889aaf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f688406e430 addr v1:172.21.15.177:6789/0 2024-09-15T04:00:57.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6889aaf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f688406e800 0x7f688406ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:57.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6889aaf640 10 monclient(hunting): picked mon.noname-b con 0x7f688406e800 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:57.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6889aaf640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6884133610 0x7f68841359f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:57.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6889aaf640 10 monclient(hunting): picked mon.noname-c con 0x7f6884133610 addr v2:172.21.15.177:3300/0 2024-09-15T04:00:57.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6889aaf640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688409b660 con 0x7f688406e430 2024-09-15T04:00:57.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6889aaf640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:57.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6889aaf640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688406c940 con 0x7f6884133610 2024-09-15T04:00:57.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6889aaf640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:57.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6889aaf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688406b0f0 con 0x7f688406e800 2024-09-15T04:00:57.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6889aaf640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:57.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6889aaf640 10 monclient(hunting): authenticate will time out at 881.526218s 2024-09-15T04:00:57.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f687bfff640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f688406e800 msgr2=0x7f688406ebf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:00:57.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f687bfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f688406e800 0x7f688406ebf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:00:57.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6883fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6884133610 0x7f68841359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:57.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6883fff640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6884133610 0x7f68841359f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.110:35110/0 (socket says 172.21.15.110:35110) 2024-09-15T04:00:57.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6883fff640 1 -- 172.21.15.110:0/4270159002 learned_addr learned my addr 172.21.15.110:0/4270159002 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:57.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6883fff640 10 monclient(hunting): get_auth_request con 0x7f6884133610 auth_method 0 2024-09-15T04:00:57.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6883fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:57.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6883fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:57.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6883fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:57.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6883fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:57.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6883fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:57.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.882+0000 7f6883fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:57.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.884+0000 7f68837fe640 1 -- 172.21.15.110:0/4270159002 >> v1:172.21.15.177:6789/0 conn(0x7f688406e430 legacy=0x7f688406a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:57.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.885+0000 7f6883fff640 10 monclient(hunting): handle_auth_done global_id 4828 payload 274 2024-09-15T04:00:57.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.885+0000 7f6883fff640 1 -- 172.21.15.110:0/4270159002 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f688406e800 msgr2=0x7f688406ebf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:00:57.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.885+0000 7f6883fff640 1 --2- 172.21.15.110:0/4270159002 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f688406e800 0x7f688406ebf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:57.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.885+0000 7f6883fff640 1 -- 172.21.15.110:0/4270159002 >> v1:172.21.15.177:6789/0 conn(0x7f688406e430 legacy=0x7f688406a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:57.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.885+0000 7f6883fff640 10 monclient: _finish_hunting 0 2024-09-15T04:00:57.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.885+0000 7f6883fff640 1 monclient: found mon.noname-c 2024-09-15T04:00:57.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.885+0000 7f6883fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:57.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.885+0000 7f6883fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T04:00:57.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.885+0000 7f6883fff640 1 -- 172.21.15.110:0/4270159002 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688406cc20 con 0x7f6884133610 2024-09-15T04:00:57.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.887+0000 7f6883fff640 1 -- 172.21.15.110:0/4270159002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6884133610 msgr2=0x7f68841359f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:57.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6883fff640 1 --2- 172.21.15.110:0/4270159002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6884133610 0x7f68841359f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f687c00d220 tx=0x7f687c00ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f026af7337200b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:57.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6882ffd640 1 -- 172.21.15.110:0/4270159002 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c009610 con 0x7f6884133610 2024-09-15T04:00:57.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6882ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:57.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6882ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:00:57.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6882ffd640 10 monclient: dump: 2024-09-15T04:00:57.889 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:57.889 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:57.889 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:57.889 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:57.889 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:57.890 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6882ffd640 10 monclient: _finish_auth 0 2024-09-15T04:00:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6882ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6882ffd640 1 -- 172.21.15.110:0/4270159002 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687c009770 con 0x7f6884133610 2024-09-15T04:00:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6889aaf640 5 monclient: authenticate success, global_id 4828 2024-09-15T04:00:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6889aaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:00:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6882ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:57.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6882ffd640 1 -- 172.21.15.110:0/4270159002 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c009b50 con 0x7f6884133610 2024-09-15T04:00:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6882ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6882ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:00:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6882ffd640 10 monclient: dump: 2024-09-15T04:00:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:57.891 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:57.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6889aaf640 10 monclient: get_monmap_and_config success 2024-09-15T04:00:57.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6889aaf640 4 set_mon_vals no callback set 2024-09-15T04:00:57.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6889aaf640 10 monclient: shutdown 2024-09-15T04:00:57.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6889aaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:57.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6889aaf640 1 -- 172.21.15.110:0/4270159002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6884133610 msgr2=0x7f68841359f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6889aaf640 1 --2- 172.21.15.110:0/4270159002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6884133610 0x7f68841359f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f687c00d220 tx=0x7f687c00ba50 comp rx=0 tx=0).stop 2024-09-15T04:00:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6889aaf640 1 -- 172.21.15.110:0/4270159002 shutdown_connections 2024-09-15T04:00:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.888+0000 7f6889aaf640 1 --2- 172.21.15.110:0/4270159002 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6884133610 0x7f68841359f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.889+0000 7f6889aaf640 1 --2- 172.21.15.110:0/4270159002 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f688406e800 0x7f688406ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.889+0000 7f6889aaf640 1 -- 172.21.15.110:0/4270159002 >> 172.21.15.110:0/4270159002 conn(0x7f688409d8f0 msgr2=0x7f688409dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.889+0000 7f6889aaf640 1 -- 172.21.15.110:0/4270159002 shutdown_connections 2024-09-15T04:00:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.889+0000 7f6889aaf640 1 -- 172.21.15.110:0/4270159002 wait complete. 2024-09-15T04:00:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.889+0000 7f6889aaf640 10 monclient: build_initial_monmap 2024-09-15T04:00:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.889+0000 7f6889aaf640 10 monclient: monmap: 2024-09-15T04:00:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:00:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:00:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:00:57.897 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:00:57.898 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:00:57.898 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:57.898 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:00:57.898 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:00:57.898 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:00:57.898 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:57.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.889+0000 7f6889aaf640 1 Processor -- start 2024-09-15T04:00:57.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.889+0000 7f6889aaf640 1 -- start start 2024-09-15T04:00:57.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.889+0000 7f6889aaf640 10 monclient: init 2024-09-15T04:00:57.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 10 monclient: _reopen_session rank -1 2024-09-15T04:00:57.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T04:00:57.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f688406e430 0x7f68841323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:57.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 10 monclient(hunting): picked mon.noname-c con 0x7f688406e430 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:00:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f688406e800 0x7f688411bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 10 monclient(hunting): picked mon.noname-a con 0x7f688406e800 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:00:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6884133610 0x7f688411c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 10 monclient(hunting): picked mon.noname-b con 0x7f6884133610 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:00:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688411fb50 con 0x7f688406e800 2024-09-15T04:00:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688409ae80 con 0x7f6884133610 2024-09-15T04:00:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 10 monclient(hunting): start opening mon connection 2024-09-15T04:00:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688406d8d0 con 0x7f688406e430 2024-09-15T04:00:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 10 monclient(hunting): _renew_subs 2024-09-15T04:00:57.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6889aaf640 10 monclient(hunting): authenticate will time out at 881.534313s 2024-09-15T04:00:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f68837fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f688406e430 msgr2=0x7f68841323e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:00:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f68837fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f688406e430 0x7f68841323e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:00:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f688406e800 0x7f688411bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f688406e800 0x7f688411bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54076/0 (socket says 172.21.15.110:54076) 2024-09-15T04:00:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 1 -- 172.21.15.110:0/3189848498 learned_addr learned my addr 172.21.15.110:0/3189848498 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:00:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 10 monclient(hunting): get_auth_request con 0x7f688406e800 auth_method 0 2024-09-15T04:00:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:00:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:00:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:00:57.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6883fff640 1 --2- 172.21.15.110:0/3189848498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6884133610 0x7f688411c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6883fff640 10 monclient(hunting): get_auth_request con 0x7f6884133610 auth_method 0 2024-09-15T04:00:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6883fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:00:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6883fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:00:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f6883fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:00:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 10 monclient(hunting): handle_auth_done global_id 54177 payload 824 2024-09-15T04:00:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 1 -- 172.21.15.110:0/3189848498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f688406e430 msgr2=0x7f68841323e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:00:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 1 --2- 172.21.15.110:0/3189848498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f688406e430 0x7f68841323e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 1 -- 172.21.15.110:0/3189848498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6884133610 msgr2=0x7f688411c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 1 --2- 172.21.15.110:0/3189848498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6884133610 0x7f688411c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 10 monclient: _finish_hunting 0 2024-09-15T04:00:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 1 monclient: found mon.noname-a 2024-09-15T04:00:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:00:57.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:00:57.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.890+0000 7f687bfff640 1 -- 172.21.15.110:0/3189848498 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c00bcf0 con 0x7f688406e800 2024-09-15T04:00:57.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.892+0000 7f687bfff640 1 -- 172.21.15.110:0/3189848498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f688406e800 msgr2=0x7f688411bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:00:57.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.892+0000 7f687bfff640 1 --2- 172.21.15.110:0/3189848498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f688406e800 0x7f688411bee0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f686c00b640 tx=0x7f686c006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:57.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f687b7fe640 1 -- 172.21.15.110:0/3189848498 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c015070 con 0x7f688406e800 2024-09-15T04:00:57.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f687b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:57.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f687b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:00:57.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f687b7fe640 10 monclient: dump: 2024-09-15T04:00:57.902 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:57.902 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:57.902 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:57.902 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:57.902 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:57.903 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f687b7fe640 10 monclient: _finish_auth 0 2024-09-15T04:00:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f687b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:00:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f687b7fe640 1 -- 172.21.15.110:0/3189848498 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686c003d60 con 0x7f688406e800 2024-09-15T04:00:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f687b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:00:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f687b7fe640 1 -- 172.21.15.110:0/3189848498 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c006380 con 0x7f688406e800 2024-09-15T04:00:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f687b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:00:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f687b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:00:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f687b7fe640 10 monclient: dump: 2024-09-15T04:00:57.903 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:00:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:00:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:00:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:00:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:00:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:00:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:00:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:00:57.904 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:00:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f6889aaf640 5 monclient: authenticate success, global_id 54177 2024-09-15T04:00:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f6889aaf640 10 monclient: _renew_subs 2024-09-15T04:00:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f6889aaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:57.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f68827fc640 4 set_mon_vals no callback set 2024-09-15T04:00:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f6889aaf640 1 -- 172.21.15.110:0/3189848498 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f688411dee0 con 0x7f688406e800 2024-09-15T04:00:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f6889aaf640 10 monclient: _renew_subs 2024-09-15T04:00:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f6889aaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.893+0000 7f6889aaf640 1 -- 172.21.15.110:0/3189848498 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688411e5a0 con 0x7f688406e800 2024-09-15T04:00:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.894+0000 7f687b7fe640 1 -- 172.21.15.110:0/3189848498 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f686c010440 con 0x7f688406e800 2024-09-15T04:00:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.894+0000 7f6889aaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.894+0000 7f6889aaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.894+0000 7f6889aaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.894+0000 7f6889aaf640 1 -- 172.21.15.110:0/3189848498 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6884070950 con 0x7f688406e800 2024-09-15T04:00:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.894+0000 7f687b7fe640 1 --2- 172.21.15.110:0/3189848498 >> v2:172.21.15.177:6812/28201 conn(0x7f68640386b0 0x7f686403ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:00:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.894+0000 7f687b7fe640 1 -- 172.21.15.110:0/3189848498 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f686c04b1c0 con 0x7f688406e800 2024-09-15T04:00:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.894+0000 7f68837fe640 1 --2- 172.21.15.110:0/3189848498 >> v2:172.21.15.177:6812/28201 conn(0x7f68640386b0 0x7f686403ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:00:57.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.894+0000 7f68837fe640 10 monclient: get_auth_request con 0x7f68640386b0 auth_method 0 2024-09-15T04:00:57.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.897+0000 7f68837fe640 1 --2- 172.21.15.110:0/3189848498 >> v2:172.21.15.177:6812/28201 conn(0x7f68640386b0 0x7f686403ab60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6874001410 tx=0x7f6874007a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:00:57.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.898+0000 7f687b7fe640 1 -- 172.21.15.110:0/3189848498 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f686c018af0 con 0x7f688406e800 2024-09-15T04:00:57.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.898+0000 7f687b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:00:57.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:57.898+0000 7f687b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:00:58.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.116+0000 7f6889aaf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:00:58.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.116+0000 7f6889aaf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:00:58.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.116+0000 7f6889aaf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:00:58.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.116+0000 7f6889aaf640 1 -- 172.21.15.110:0/3189848498 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6884068110 con 0x7f688406e800 2024-09-15T04:00:58.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.116+0000 7f687b7fe640 1 -- 172.21.15.110:0/3189848498 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f686c018310 con 0x7f688406e800 2024-09-15T04:00:58.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.116+0000 7f687b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:00:58.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.116+0000 7f687b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:00:58.118 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:00:58.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.118+0000 7f6889aaf640 1 -- 172.21.15.110:0/3189848498 >> v2:172.21.15.177:6812/28201 conn(0x7f68640386b0 msgr2=0x7f686403ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:58.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.118+0000 7f6889aaf640 1 --2- 172.21.15.110:0/3189848498 >> v2:172.21.15.177:6812/28201 conn(0x7f68640386b0 0x7f686403ab60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6874001410 tx=0x7f6874007a10 comp rx=0 tx=0).stop 2024-09-15T04:00:58.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.118+0000 7f6889aaf640 10 monclient: shutdown 2024-09-15T04:00:58.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.118+0000 7f6889aaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:00:58.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.118+0000 7f6889aaf640 1 -- 172.21.15.110:0/3189848498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f688406e800 msgr2=0x7f688411bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:00:58.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.118+0000 7f6889aaf640 1 --2- 172.21.15.110:0/3189848498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f688406e800 0x7f688411bee0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f686c00b640 tx=0x7f686c006f90 comp rx=0 tx=0).stop 2024-09-15T04:00:58.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.118+0000 7f6889aaf640 1 -- 172.21.15.110:0/3189848498 shutdown_connections 2024-09-15T04:00:58.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.118+0000 7f6889aaf640 1 --2- 172.21.15.110:0/3189848498 >> v2:172.21.15.177:6812/28201 conn(0x7f68640386b0 0x7f686403ab60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:58.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.118+0000 7f6889aaf640 1 --2- 172.21.15.110:0/3189848498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6884133610 0x7f688411c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:58.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.118+0000 7f6889aaf640 1 --2- 172.21.15.110:0/3189848498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f688406e800 0x7f688411bee0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:58.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.118+0000 7f6889aaf640 1 --2- 172.21.15.110:0/3189848498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f688406e430 0x7f68841323e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:00:58.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.118+0000 7f6889aaf640 1 -- 172.21.15.110:0/3189848498 >> 172.21.15.110:0/3189848498 conn(0x7f688409d8f0 msgr2=0x7f688409f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:00:58.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.118+0000 7f6889aaf640 1 -- 172.21.15.110:0/3189848498 shutdown_connections 2024-09-15T04:00:58.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:00:58.118+0000 7f6889aaf640 1 -- 172.21.15.110:0/3189848498 wait complete. 2024-09-15T04:01:01.131 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:01:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.233+0000 7fecea19d640 10 monclient: get_monmap_and_config 2024-09-15T04:01:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.233+0000 7fecea19d640 10 monclient: build_initial_monmap 2024-09-15T04:01:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.233+0000 7fecea19d640 10 monclient: monmap: 2024-09-15T04:01:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:01:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:01:01.234417+0000 2024-09-15T04:01:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:01:01.234417+0000 2024-09-15T04:01:01.234 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:01:01.235 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:01.235 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:01:01.235 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:01:01.235 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:01:01.235 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:01:01.235 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:01:01.235 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:01.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 1 Processor -- start 2024-09-15T04:01:01.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 1 -- start start 2024-09-15T04:01:01.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 10 monclient: init 2024-09-15T04:01:01.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 10 monclient: _reopen_session rank -1 2024-09-15T04:01:01.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-15T04:01:01.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fece4131ac0 addr v1:172.21.15.110:6789/0 2024-09-15T04:01:01.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fece4097150 0x7fece4136990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:01.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 10 monclient(hunting): picked mon.noname-a con 0x7fece4097150 addr v2:172.21.15.110:3300/0 2024-09-15T04:01:01.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece4136ed0 0x7fece41392b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:01.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 10 monclient(hunting): picked mon.noname-b con 0x7fece4136ed0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:01:01.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece4095ec0 con 0x7fece4131ac0 2024-09-15T04:01:01.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:01.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece41325a0 con 0x7fece4097150 2024-09-15T04:01:01.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:01.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece4095be0 con 0x7fece4136ed0 2024-09-15T04:01:01.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 10 monclient(hunting): _renew_subs 2024-09-15T04:01:01.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fecea19d640 10 monclient(hunting): authenticate will time out at 884.878779s 2024-09-15T04:01:01.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fece3fff640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece4136ed0 msgr2=0x7fece41392b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:01.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fece3fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece4136ed0 0x7fece41392b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:01:01.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fece2ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fece4097150 0x7fece4136990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:01.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fece2ffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fece4097150 0x7fece4136990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56224/0 (socket says 172.21.15.110:56224) 2024-09-15T04:01:01.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fece2ffd640 1 -- 172.21.15.110:0/2967945774 learned_addr learned my addr 172.21.15.110:0/2967945774 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:01:01.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fece2ffd640 10 monclient(hunting): get_auth_request con 0x7fece4097150 auth_method 0 2024-09-15T04:01:01.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fece2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:01.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fece2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:01.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.234+0000 7fece2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:01.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.237+0000 7fece37fe640 1 -- 172.21.15.110:0/2967945774 >> v1:172.21.15.110:6789/0 conn(0x7fece4131ac0 legacy=0x7fece4131e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:01.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.237+0000 7fece2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:01:01.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.237+0000 7fece2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:01:01.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.237+0000 7fece2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:01:01.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.237+0000 7fece2ffd640 10 monclient(hunting): handle_auth_done global_id 54180 payload 274 2024-09-15T04:01:01.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.237+0000 7fece2ffd640 1 -- 172.21.15.110:0/2967945774 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece4136ed0 msgr2=0x7fece41392b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:01:01.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.237+0000 7fece2ffd640 1 --2- 172.21.15.110:0/2967945774 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece4136ed0 0x7fece41392b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:01.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.237+0000 7fece2ffd640 1 -- 172.21.15.110:0/2967945774 >> v1:172.21.15.110:6789/0 conn(0x7fece4131ac0 legacy=0x7fece4131e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:01.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.237+0000 7fece2ffd640 10 monclient: _finish_hunting 0 2024-09-15T04:01:01.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.237+0000 7fece2ffd640 1 monclient: found mon.noname-a 2024-09-15T04:01:01.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.237+0000 7fece2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:01:01.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.237+0000 7fece2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:01:01.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.237+0000 7fece2ffd640 1 -- 172.21.15.110:0/2967945774 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece409cbd0 con 0x7fece4097150 2024-09-15T04:01:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fece2ffd640 1 -- 172.21.15.110:0/2967945774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece4097150 msgr2=0x7fece4136990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fece2ffd640 1 --2- 172.21.15.110:0/2967945774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece4097150 0x7fece4136990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fecd4009990 tx=0x7fecd40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d491d46b91ddd41b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fece27fc640 1 -- 172.21.15.110:0/2967945774 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd402f4b0 con 0x7fece4097150 2024-09-15T04:01:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fece27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fece27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:01:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fece27fc640 10 monclient: dump: 2024-09-15T04:01:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:01.245 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:01.246 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:01.246 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:01.246 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:01.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fece27fc640 10 monclient: _finish_auth 0 2024-09-15T04:01:01.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fece27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:01:01.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fece27fc640 1 -- 172.21.15.110:0/2967945774 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd402f610 con 0x7fece4097150 2024-09-15T04:01:01.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fecea19d640 5 monclient: authenticate success, global_id 54180 2024-09-15T04:01:01.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fecea19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:01:01.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fece27fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:01:01.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fecea19d640 10 monclient: get_monmap_and_config success 2024-09-15T04:01:01.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fece27fc640 1 -- 172.21.15.110:0/2967945774 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd402f9f0 con 0x7fece4097150 2024-09-15T04:01:01.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fecea19d640 4 set_mon_vals no callback set 2024-09-15T04:01:01.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fece27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:01.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fece27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:01:01.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fece27fc640 10 monclient: dump: 2024-09-15T04:01:01.247 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:01.247 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:01.247 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:01.247 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:01.247 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:01.247 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:01.247 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:01.247 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:01.247 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:01.247 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:01.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fecea19d640 10 monclient: shutdown 2024-09-15T04:01:01.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fecea19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:01:01.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fecea19d640 1 -- 172.21.15.110:0/2967945774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece4097150 msgr2=0x7fece4136990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:01.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fecea19d640 1 --2- 172.21.15.110:0/2967945774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece4097150 0x7fece4136990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fecd4009990 tx=0x7fecd40091d0 comp rx=0 tx=0).stop 2024-09-15T04:01:01.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.240+0000 7fecea19d640 1 -- 172.21.15.110:0/2967945774 shutdown_connections 2024-09-15T04:01:01.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.241+0000 7fecea19d640 1 --2- 172.21.15.110:0/2967945774 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece4136ed0 0x7fece41392b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:01.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.241+0000 7fecea19d640 1 --2- 172.21.15.110:0/2967945774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece4097150 0x7fece4136990 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:01.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.241+0000 7fecea19d640 1 -- 172.21.15.110:0/2967945774 >> 172.21.15.110:0/2967945774 conn(0x7fece40a1440 msgr2=0x7fece40a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:01:01.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.241+0000 7fecea19d640 1 -- 172.21.15.110:0/2967945774 shutdown_connections 2024-09-15T04:01:01.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.241+0000 7fecea19d640 1 -- 172.21.15.110:0/2967945774 wait complete. 2024-09-15T04:01:01.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.241+0000 7fecea19d640 10 monclient: build_initial_monmap 2024-09-15T04:01:01.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.241+0000 7fecea19d640 10 monclient: monmap: 2024-09-15T04:01:01.248 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:01:01.248 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:01:01.248 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:01:01.249 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:01:01.249 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:01:01.249 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:01.249 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:01:01.249 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:01:01.249 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:01:01.249 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:01.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 1 Processor -- start 2024-09-15T04:01:01.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 1 -- start start 2024-09-15T04:01:01.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 10 monclient: init 2024-09-15T04:01:01.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 10 monclient: _reopen_session rank -1 2024-09-15T04:01:01.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T04:01:01.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fece4097150 0x7fece41307e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 10 monclient(hunting): picked mon.noname-b con 0x7fece4097150 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece4131ac0 0x7fece4119370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 10 monclient(hunting): picked mon.noname-c con 0x7fece4131ac0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece4136ed0 0x7fece41198b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 10 monclient(hunting): picked mon.noname-a con 0x7fece4136ed0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece411cfb0 con 0x7fece4136ed0 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece41421f0 con 0x7fece4097150 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece411b340 con 0x7fece4131ac0 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 10 monclient(hunting): _renew_subs 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fecea19d640 10 monclient(hunting): authenticate will time out at 884.886557s 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece2ffd640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece4131ac0 msgr2=0x7fece4119370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:01.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece2ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece4131ac0 0x7fece4119370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:01:01.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece4136ed0 0x7fece41198b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:01.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece4136ed0 0x7fece41198b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56230/0 (socket says 172.21.15.110:56230) 2024-09-15T04:01:01.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 1 -- 172.21.15.110:0/2932915479 learned_addr learned my addr 172.21.15.110:0/2932915479 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:01:01.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 10 monclient(hunting): get_auth_request con 0x7fece4136ed0 auth_method 0 2024-09-15T04:01:01.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:01.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:01.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:01.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:01:01.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:01:01.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:01:01.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 10 monclient(hunting): handle_auth_done global_id 54183 payload 824 2024-09-15T04:01:01.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece37fe640 1 --2- 172.21.15.110:0/2932915479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fece4097150 0x7fece41307e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:01.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 1 -- 172.21.15.110:0/2932915479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece4131ac0 msgr2=0x7fece4119370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:01:01.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 1 --2- 172.21.15.110:0/2932915479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece4131ac0 0x7fece4119370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:01.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 1 -- 172.21.15.110:0/2932915479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fece4097150 msgr2=0x7fece41307e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:01.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 1 --2- 172.21.15.110:0/2932915479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fece4097150 0x7fece41307e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:01.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 10 monclient: _finish_hunting 0 2024-09-15T04:01:01.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 1 monclient: found mon.noname-a 2024-09-15T04:01:01.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:01:01.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:01:01.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.242+0000 7fece3fff640 1 -- 172.21.15.110:0/2932915479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd4009450 con 0x7fece4136ed0 2024-09-15T04:01:01.252 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.243+0000 7fece37fe640 10 monclient: get_auth_request con 0x7fece4097150 auth_method 0 2024-09-15T04:01:01.253 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.243+0000 7fece37fe640 1 --2- 172.21.15.110:0/2932915479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fece4097150 0x7fece41307e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:01:01.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7fece3fff640 1 -- 172.21.15.110:0/2932915479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece4136ed0 msgr2=0x7fece41198b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:01.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7fece3fff640 1 --2- 172.21.15.110:0/2932915479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece4136ed0 0x7fece41198b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fecdc00beb0 tx=0x7fecdc0096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:01.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7feccb7fe640 1 -- 172.21.15.110:0/2932915479 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc016430 con 0x7fece4136ed0 2024-09-15T04:01:01.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7feccb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7feccb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:01:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7feccb7fe640 10 monclient: dump: 2024-09-15T04:01:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:01.256 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7feccb7fe640 10 monclient: _finish_auth 0 2024-09-15T04:01:01.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7feccb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:01:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7feccb7fe640 1 -- 172.21.15.110:0/2932915479 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecdc0165b0 con 0x7fece4136ed0 2024-09-15T04:01:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7fecea19d640 5 monclient: authenticate success, global_id 54183 2024-09-15T04:01:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7fecea19d640 10 monclient: _renew_subs 2024-09-15T04:01:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7fecea19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:01:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7fecea19d640 1 -- 172.21.15.110:0/2932915479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece409d560 con 0x7fece4136ed0 2024-09-15T04:01:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7feccb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:01:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7feccb7fe640 1 -- 172.21.15.110:0/2932915479 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecdc016760 con 0x7fece4136ed0 2024-09-15T04:01:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7feccb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7feccb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:01:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7feccb7fe640 10 monclient: dump: 2024-09-15T04:01:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:01.257 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:01.258 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:01.258 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:01.258 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:01.258 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:01.258 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:01.258 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:01.258 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:01.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7fecea19d640 10 monclient: _renew_subs 2024-09-15T04:01:01.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7fecea19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:01:01.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7fecea19d640 1 -- 172.21.15.110:0/2932915479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece411ae40 con 0x7fece4136ed0 2024-09-15T04:01:01.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.245+0000 7fece1ffb640 4 set_mon_vals no callback set 2024-09-15T04:01:01.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.246+0000 7feccb7fe640 1 -- 172.21.15.110:0/2932915479 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fecdc012020 con 0x7fece4136ed0 2024-09-15T04:01:01.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.246+0000 7fecea19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:01:01.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.246+0000 7fecea19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:01:01.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.246+0000 7fecea19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:01:01.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.246+0000 7fecea19d640 1 -- 172.21.15.110:0/2932915479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece411bab0 con 0x7fece4136ed0 2024-09-15T04:01:01.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.246+0000 7feccb7fe640 1 --2- 172.21.15.110:0/2932915479 >> v2:172.21.15.177:6812/28201 conn(0x7fecd00386b0 0x7fecd003ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:01.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.246+0000 7feccb7fe640 1 -- 172.21.15.110:0/2932915479 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fecdc04b8f0 con 0x7fece4136ed0 2024-09-15T04:01:01.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.246+0000 7fece37fe640 1 --2- 172.21.15.110:0/2932915479 >> v2:172.21.15.177:6812/28201 conn(0x7fecd00386b0 0x7fecd003ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:01.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.246+0000 7fece37fe640 10 monclient: get_auth_request con 0x7fecd00386b0 auth_method 0 2024-09-15T04:01:01.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.249+0000 7fece37fe640 1 --2- 172.21.15.110:0/2932915479 >> v2:172.21.15.177:6812/28201 conn(0x7fecd00386b0 0x7fecd003ab60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fece4096050 tx=0x7fecd8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:01.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.251+0000 7feccb7fe640 1 -- 172.21.15.110:0/2932915479 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fecdc0138e0 con 0x7fece4136ed0 2024-09-15T04:01:01.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.251+0000 7feccb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:01:01.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.251+0000 7feccb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:01:01.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.464+0000 7fecea19d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:01:01.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.464+0000 7fecea19d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:01:01.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.464+0000 7fecea19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:01:01.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.464+0000 7fecea19d640 1 -- 172.21.15.110:0/2932915479 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fece4070950 con 0x7fece4136ed0 2024-09-15T04:01:01.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.464+0000 7feccb7fe640 1 -- 172.21.15.110:0/2932915479 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1540 (secure 0 0 0) 0x7fecdc01d300 con 0x7fece4136ed0 2024-09-15T04:01:01.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.464+0000 7feccb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:01:01.465 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.464+0000 7feccb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:01:01.465 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:01:01.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.466+0000 7fecea19d640 1 -- 172.21.15.110:0/2932915479 >> v2:172.21.15.177:6812/28201 conn(0x7fecd00386b0 msgr2=0x7fecd003ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:01.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.466+0000 7fecea19d640 1 --2- 172.21.15.110:0/2932915479 >> v2:172.21.15.177:6812/28201 conn(0x7fecd00386b0 0x7fecd003ab60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fece4096050 tx=0x7fecd8009040 comp rx=0 tx=0).stop 2024-09-15T04:01:01.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.466+0000 7fecea19d640 10 monclient: shutdown 2024-09-15T04:01:01.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.466+0000 7fecea19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:01:01.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.466+0000 7fecea19d640 1 -- 172.21.15.110:0/2932915479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece4136ed0 msgr2=0x7fece41198b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:01.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.466+0000 7fecea19d640 1 --2- 172.21.15.110:0/2932915479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece4136ed0 0x7fece41198b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fecdc00beb0 tx=0x7fecdc0096d0 comp rx=0 tx=0).stop 2024-09-15T04:01:01.467 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.466+0000 7fecea19d640 1 -- 172.21.15.110:0/2932915479 shutdown_connections 2024-09-15T04:01:01.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.466+0000 7fecea19d640 1 --2- 172.21.15.110:0/2932915479 >> v2:172.21.15.177:6812/28201 conn(0x7fecd00386b0 0x7fecd003ab60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:01.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.466+0000 7fecea19d640 1 --2- 172.21.15.110:0/2932915479 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece4136ed0 0x7fece41198b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:01.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.466+0000 7fecea19d640 1 --2- 172.21.15.110:0/2932915479 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fece4131ac0 0x7fece4119370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:01.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.466+0000 7fecea19d640 1 --2- 172.21.15.110:0/2932915479 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fece4097150 0x7fece41307e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:01.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.466+0000 7fecea19d640 1 -- 172.21.15.110:0/2932915479 >> 172.21.15.110:0/2932915479 conn(0x7fece40a1440 msgr2=0x7fece409edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:01:01.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.466+0000 7fecea19d640 1 -- 172.21.15.110:0/2932915479 shutdown_connections 2024-09-15T04:01:01.468 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:01.466+0000 7fecea19d640 1 -- 172.21.15.110:0/2932915479 wait complete. 2024-09-15T04:01:04.478 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:01:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.582+0000 7f5d5079d640 10 monclient: get_monmap_and_config 2024-09-15T04:01:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.582+0000 7f5d5079d640 10 monclient: build_initial_monmap 2024-09-15T04:01:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.582+0000 7f5d5079d640 10 monclient: monmap: 2024-09-15T04:01:04.583 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:01:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:01:04.583119+0000 2024-09-15T04:01:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:01:04.583119+0000 2024-09-15T04:01:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:01:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:01:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:01:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:01:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:01:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:01:04.584 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.582+0000 7f5d5079d640 1 Processor -- start 2024-09-15T04:01:04.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.582+0000 7f5d5079d640 1 -- start start 2024-09-15T04:01:04.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.582+0000 7f5d5079d640 10 monclient: init 2024-09-15T04:01:04.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d5079d640 10 monclient: _reopen_session rank -1 2024-09-15T04:01:04.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d5079d640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-15T04:01:04.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d5079d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:04.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d5079d640 10 monclient(hunting): picked mon.noname-a con 0x7f5d480a1a60 addr v2:172.21.15.110:3300/0 2024-09-15T04:01:04.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d5079d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d4809f870 addr v1:172.21.15.110:6789/0 2024-09-15T04:01:04.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d5079d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d48138b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:04.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d5079d640 10 monclient(hunting): picked mon.noname-c con 0x7f5d480a0380 addr v2:172.21.15.177:3300/0 2024-09-15T04:01:04.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d5079d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d480a1260 con 0x7f5d4809f870 2024-09-15T04:01:04.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d5079d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:04.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d5079d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d4809cdf0 con 0x7f5d480a1a60 2024-09-15T04:01:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d5079d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d5079d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d4809c870 con 0x7f5d480a0380 2024-09-15T04:01:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d5079d640 10 monclient(hunting): _renew_subs 2024-09-15T04:01:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d5079d640 10 monclient(hunting): authenticate will time out at 888.227466s 2024-09-15T04:01:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d4e539640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:04.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d4e539640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:56234/0 (socket says 172.21.15.110:56234) 2024-09-15T04:01:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 learned_addr learned my addr 172.21.15.110:0/1236941760 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:01:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d4e539640 10 monclient(hunting): get_auth_request con 0x7f5d480a1a60 auth_method 0 2024-09-15T04:01:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d4e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d4e539640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d4e539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d48138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d4ed3a640 10 monclient(hunting): get_auth_request con 0x7f5d480a0380 auth_method 0 2024-09-15T04:01:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d4ed3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d4ed3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:04.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.583+0000 7f5d4ed3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:04.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.584+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d48138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T04:01:04.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.584+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d48138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:04.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.584+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d48138b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:04.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.584+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d48138b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:01:04.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.585+0000 7f5d4dd38640 1 -- 172.21.15.110:0/1236941760 >> v1:172.21.15.110:6789/0 conn(0x7f5d4809f870 legacy=0x7f5d4809fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:04.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.585+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 msgr2=0x7f5d4809d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T04:01:04.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.585+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 msgr2=0x7f5d4809d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:04.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.585+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:04.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.585+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:01:04.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.785+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d48138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:04.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.785+0000 7f5d4ed3a640 10 monclient(hunting): get_auth_request con 0x7f5d480a0380 auth_method 0 2024-09-15T04:01:04.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.785+0000 7f5d4ed3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:04.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.785+0000 7f5d4ed3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:04.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.785+0000 7f5d4ed3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:01:04.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.785+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d48138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T04:01:04.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.785+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d48138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:04.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.785+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d48138b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:04.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.785+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d48138b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T04:01:04.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.786+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:04.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.786+0000 7f5d4e539640 10 monclient(hunting): get_auth_request con 0x7f5d480a1a60 auth_method 0 2024-09-15T04:01:04.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.786+0000 7f5d4e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:04.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.786+0000 7f5d4e539640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:04.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.786+0000 7f5d4e539640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:01:04.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.786+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 msgr2=0x7f5d4809d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T04:01:04.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.786+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 msgr2=0x7f5d4809d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:04.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.786+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:04.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:04.786+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T04:01:05.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.186+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d48138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:05.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.186+0000 7f5d4ed3a640 10 monclient(hunting): get_auth_request con 0x7f5d480a0380 auth_method 0 2024-09-15T04:01:05.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.186+0000 7f5d4ed3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:05.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.186+0000 7f5d4ed3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:05.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.186+0000 7f5d4ed3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:01:05.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.186+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d48138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T04:01:05.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.186+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d48138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:05.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.186+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d48138b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:05.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.186+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d48138b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T04:01:05.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.187+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:05.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.187+0000 7f5d4e539640 10 monclient(hunting): get_auth_request con 0x7f5d480a1a60 auth_method 0 2024-09-15T04:01:05.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.187+0000 7f5d4e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:05.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.187+0000 7f5d4e539640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:05.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.187+0000 7f5d4e539640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:01:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.187+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 msgr2=0x7f5d4809d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T04:01:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.187+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 msgr2=0x7f5d4809d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.187+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:05.191 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.187+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T04:01:05.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.987+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d48138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:05.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.987+0000 7f5d4ed3a640 10 monclient(hunting): get_auth_request con 0x7f5d480a0380 auth_method 0 2024-09-15T04:01:05.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4ed3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:05.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4ed3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:05.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4ed3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:01:05.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d48138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T04:01:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d48138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d48138b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d48138b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T04:01:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4e539640 10 monclient(hunting): get_auth_request con 0x7f5d480a1a60 auth_method 0 2024-09-15T04:01:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4e539640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4e539640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:01:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 msgr2=0x7f5d4809d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T04:01:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 msgr2=0x7f5d4809d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:05.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:05.988+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T04:01:07.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 10 monclient(hunting): tick 2024-09-15T04:01:07.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 1 monclient(hunting): continuing hunt 2024-09-15T04:01:07.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T04:01:07.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d48138b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:01:07.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d48138b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:07.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 msgr2=0x7f5d4809d4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:01:07.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d480a1a60 0x7f5d4809d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 >> v1:172.21.15.110:6789/0 conn(0x7f5d4809f870 legacy=0x7f5d4809fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 10 monclient: _add_conns ranks=[1,2] 2024-09-15T04:01:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d340036d0 addr v1:172.21.15.177:6789/0 2024-09-15T04:01:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 0x7f5d34009860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 10 monclient(hunting): picked mon.noname-b con 0x7f5d340073b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:01:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d480a0c60 con 0x7f5d340036d0 2024-09-15T04:01:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4809c870 con 0x7f5d340073b0 2024-09-15T04:01:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d47fff640 10 monclient(hunting): _renew_subs 2024-09-15T04:01:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d4dd38640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 msgr2=0x7f5d34009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:07.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.583+0000 7f5d4dd38640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 0x7f5d34009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:01:07.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.586+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v1:172.21.15.177:6789/0 conn(0x7f5d340036d0 legacy=0x7f5d34005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.784+0000 7f5d4dd38640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 msgr2=0x7f5d34009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:07.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:07.784+0000 7f5d4dd38640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 0x7f5d34009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T04:01:08.185 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:08.185+0000 7f5d4dd38640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 msgr2=0x7f5d34009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:08.186 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:08.185+0000 7f5d4dd38640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 0x7f5d34009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T04:01:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:08.986+0000 7f5d4dd38640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 msgr2=0x7f5d34009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:08.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:08.986+0000 7f5d4dd38640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 0x7f5d34009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T04:01:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 10 monclient(hunting): tick 2024-09-15T04:01:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 1 monclient(hunting): continuing hunt 2024-09-15T04:01:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T04:01:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 msgr2=0x7f5d34009860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:01:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 0x7f5d34009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 >> v1:172.21.15.177:6789/0 conn(0x7f5d340036d0 legacy=0x7f5d34005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-15T04:01:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 reap_dead start 2024-09-15T04:01:10.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d3400a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 10 monclient(hunting): picked mon.noname-c con 0x7f5d480a0380 addr v2:172.21.15.177:3300/0 2024-09-15T04:01:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d4809f870 addr v1:172.21.15.110:6789/0 2024-09-15T04:01:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 0x7f5d34009860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 10 monclient(hunting): picked mon.noname-b con 0x7f5d340073b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:01:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3400b030 con 0x7f5d4809f870 2024-09-15T04:01:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d4809c870 con 0x7f5d480a0380 2024-09-15T04:01:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d34006310 con 0x7f5d340073b0 2024-09-15T04:01:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d47fff640 10 monclient(hunting): _renew_subs 2024-09-15T04:01:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 msgr2=0x7f5d34009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:10.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 0x7f5d34009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:01:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d3400a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d4e539640 10 monclient(hunting): get_auth_request con 0x7f5d480a0380 auth_method 0 2024-09-15T04:01:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d4e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d4e539640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d4e539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d3400a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:01:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d3400a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d3400a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.584+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d3400a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:01:10.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.586+0000 7f5d4dd38640 1 -- 172.21.15.110:0/1236941760 >> v1:172.21.15.110:6789/0 conn(0x7f5d4809f870 legacy=0x7f5d3400a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:10.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.785+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 msgr2=0x7f5d34009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:10.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.785+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 0x7f5d34009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T04:01:10.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.785+0000 7f5d4d537640 1 -- 172.21.15.110:0/1236941760 DispatchQueue::entry inject delay of 0.002000 2024-09-15T04:01:10.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.785+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d3400a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:10.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.785+0000 7f5d4e539640 10 monclient(hunting): get_auth_request con 0x7f5d480a0380 auth_method 0 2024-09-15T04:01:10.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.785+0000 7f5d4e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:10.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.785+0000 7f5d4e539640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:10.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.785+0000 7f5d4e539640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:01:10.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.785+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d3400a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T04:01:10.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.785+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d3400a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:10.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.785+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d3400a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:10.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:10.785+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d3400a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T04:01:11.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.186+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 msgr2=0x7f5d34009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:11.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.186+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 0x7f5d34009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T04:01:11.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.186+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d3400a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:11.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.186+0000 7f5d4e539640 10 monclient(hunting): get_auth_request con 0x7f5d480a0380 auth_method 0 2024-09-15T04:01:11.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.186+0000 7f5d4e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:11.189 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.186+0000 7f5d4e539640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:11.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.186+0000 7f5d4e539640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:01:11.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.186+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d3400a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:01:11.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.186+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d3400a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:11.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.186+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d3400a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:11.190 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.186+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d3400a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T04:01:11.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.987+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 msgr2=0x7f5d34009860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:11.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.987+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 0x7f5d34009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T04:01:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.988+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d3400a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.988+0000 7f5d4e539640 10 monclient(hunting): get_auth_request con 0x7f5d480a0380 auth_method 0 2024-09-15T04:01:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.988+0000 7f5d4e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.988+0000 7f5d4e539640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.988+0000 7f5d4e539640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:01:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.988+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d3400a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:01:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.988+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d3400a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.988+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d3400a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:11.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:11.988+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d3400a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T04:01:13.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 10 monclient(hunting): tick 2024-09-15T04:01:13.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 1 monclient(hunting): continuing hunt 2024-09-15T04:01:13.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T04:01:13.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 msgr2=0x7f5d34009860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:01:13.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d340073b0 0x7f5d34009860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:13.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 msgr2=0x7f5d3400a150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:01:13.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.177:3300/0 conn(0x7f5d480a0380 0x7f5d3400a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:13.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 >> v1:172.21.15.110:6789/0 conn(0x7f5d4809f870 legacy=0x7f5d3400a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:13.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 10 monclient: _add_conns ranks=[1,3] 2024-09-15T04:01:13.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d340067d0 addr v1:172.21.15.177:6789/0 2024-09-15T04:01:13.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 0x7f5d34013350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:13.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 10 monclient(hunting): picked mon.noname-a con 0x7f5d34006c40 addr v2:172.21.15.110:3300/0 2024-09-15T04:01:13.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d340064a0 con 0x7f5d340067d0 2024-09-15T04:01:13.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:13.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 1 -- 172.21.15.110:0/1236941760 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d4809c870 con 0x7f5d34006c40 2024-09-15T04:01:13.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d47fff640 10 monclient(hunting): _renew_subs 2024-09-15T04:01:13.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d4dd38640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 0x7f5d34013350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:13.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d4dd38640 10 monclient(hunting): get_auth_request con 0x7f5d34006c40 auth_method 0 2024-09-15T04:01:13.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d4dd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:13.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d4dd38640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:13.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d4dd38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:13.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d4dd38640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 msgr2=0x7f5d34013350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:01:13.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d4dd38640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 msgr2=0x7f5d34013350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:13.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d4dd38640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 0x7f5d34013350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:13.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.584+0000 7f5d4dd38640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 0x7f5d34013350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:01:13.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.587+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> v1:172.21.15.177:6789/0 conn(0x7f5d340067d0 legacy=0x7f5d34005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:13.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.785+0000 7f5d4dd38640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 0x7f5d34013350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:13.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.785+0000 7f5d4dd38640 10 monclient(hunting): get_auth_request con 0x7f5d34006c40 auth_method 0 2024-09-15T04:01:13.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.785+0000 7f5d4dd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:13.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.785+0000 7f5d4dd38640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:13.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.785+0000 7f5d4dd38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:01:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.785+0000 7f5d4dd38640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 msgr2=0x7f5d34013350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:01:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.785+0000 7f5d4dd38640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 msgr2=0x7f5d34013350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.785+0000 7f5d4dd38640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 0x7f5d34013350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:13.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:13.785+0000 7f5d4dd38640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 0x7f5d34013350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T04:01:14.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.186+0000 7f5d4dd38640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 0x7f5d34013350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.186+0000 7f5d4dd38640 10 monclient(hunting): get_auth_request con 0x7f5d34006c40 auth_method 0 2024-09-15T04:01:14.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.186+0000 7f5d4dd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.187 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.186+0000 7f5d4dd38640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.186+0000 7f5d4dd38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:01:14.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.186+0000 7f5d4dd38640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 msgr2=0x7f5d34013350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:01:14.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.186+0000 7f5d4dd38640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 msgr2=0x7f5d34013350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:01:14.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.186+0000 7f5d4dd38640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 0x7f5d34013350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:01:14.188 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.186+0000 7f5d4dd38640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 0x7f5d34013350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T04:01:14.198 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.198+0000 7f5d4d537640 1 -- 172.21.15.110:0/1236941760 <== mon.1 v1:172.21.15.177:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295409075 0 0) 0x7f5d340064a0 con 0x7f5d340067d0 2024-09-15T04:01:14.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.198+0000 7f5d4d537640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.198+0000 7f5d4d537640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:14.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.198+0000 7f5d4d537640 10 monclient(hunting): my global_id is 4834 2024-09-15T04:01:14.199 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.198+0000 7f5d4d537640 1 -- 172.21.15.110:0/1236941760 --> v1:172.21.15.177:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d300034e0 con 0x7f5d340067d0 2024-09-15T04:01:14.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 1 -- 172.21.15.110:0/1236941760 <== mon.1 v1:172.21.15.177:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 92118095 0 0) 0x7f5d300034e0 con 0x7f5d340067d0 2024-09-15T04:01:14.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 1 -- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 msgr2=0x7f5d34013350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:01:14.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 1 --2- 172.21.15.110:0/1236941760 >> v2:172.21.15.110:3300/0 conn(0x7f5d34006c40 0x7f5d34013350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 10 monclient: _finish_hunting 0 2024-09-15T04:01:14.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 1 monclient: found mon.noname-c-legacy 2024-09-15T04:01:14.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:01:14.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.177:6789/0 2024-09-15T04:01:14.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 1 -- 172.21.15.110:0/1236941760 --> v1:172.21.15.177:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d340148a0 con 0x7f5d340067d0 2024-09-15T04:01:14.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 10 monclient: _finish_auth 0 2024-09-15T04:01:14.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:01:14.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 1 -- 172.21.15.110:0/1236941760 <== mon.1 v1:172.21.15.177:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3165669920 0 0) 0x7f5d4000c7f0 con 0x7f5d340067d0 2024-09-15T04:01:14.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-15T04:01:14.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 10 monclient: dump: 2024-09-15T04:01:14.216 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.216 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.216 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.216 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.216 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.216 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.217 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] but i'm connected to v1:172.21.15.177:6789/0, reconnecting 2024-09-15T04:01:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 10 monclient: _reopen_session rank -1 2024-09-15T04:01:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.211+0000 7f5d4d537640 1 -- 172.21.15.110:0/1236941760 >> v1:172.21.15.177:6789/0 conn(0x7f5d340067d0 legacy=0x7f5d34005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T04:01:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 reap_dead start 2024-09-15T04:01:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d30003a00 0x7f5d30003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 10 monclient(hunting): picked mon.a con 0x7f5d30003a00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:01:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d30004780 0x7f5d30004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 10 monclient(hunting): picked mon.b con 0x7f5d30004780 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:01:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d30005180 0x7f5d3000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 10 monclient(hunting): picked mon.c con 0x7f5d30005180 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:01:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 1 -- 172.21.15.110:0/1236941760 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4809c870 con 0x7f5d30003a00 2024-09-15T04:01:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 1 -- 172.21.15.110:0/1236941760 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3000c120 con 0x7f5d30004780 2024-09-15T04:01:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 1 -- 172.21.15.110:0/1236941760 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3000c2e0 con 0x7f5d30005180 2024-09-15T04:01:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 10 monclient(hunting): _renew_subs 2024-09-15T04:01:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4d537640 10 monclient(hunting): _finish_auth 0 2024-09-15T04:01:14.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d30005180 msgr2=0x7f5d3000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d30005180 0x7f5d3000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:01:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d30003a00 0x7f5d30003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 10 monclient(hunting): get_auth_request con 0x7f5d30003a00 auth_method 0 2024-09-15T04:01:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:01:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:01:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:01:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:01:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4dd38640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d30004780 0x7f5d30004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4dd38640 10 monclient(hunting): get_auth_request con 0x7f5d30004780 auth_method 0 2024-09-15T04:01:14.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4dd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4dd38640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4dd38640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:01:14.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 10 monclient(hunting): handle_auth_done global_id 4834 payload 293 2024-09-15T04:01:14.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d30005180 msgr2=0x7f5d3000b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:01:14.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d30005180 0x7f5d3000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d30004780 msgr2=0x7f5d30004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:01:14.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d30004780 0x7f5d30004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 10 monclient: _finish_hunting 0 2024-09-15T04:01:14.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 1 monclient: found mon.a 2024-09-15T04:01:14.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:01:14.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:01:14.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d300034e0 con 0x7f5d30003a00 2024-09-15T04:01:14.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 10 monclient: _finish_auth 0 2024-09-15T04:01:14.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d4e539640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:01:14.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d5079d640 5 monclient: authenticate success, global_id 4834 2024-09-15T04:01:14.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.212+0000 7f5d5079d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:01:14.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d4e539640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d30003a00 msgr2=0x7f5d30003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:14.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d30003a00 0x7f5d30003df0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5d40006f60 tx=0x7f5d40008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81fd61303bfbf8c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:14.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d4d537640 1 -- 172.21.15.110:0/1236941760 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4002cce0 con 0x7f5d30003a00 2024-09-15T04:01:14.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d4d537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d4d537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:01:14.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d4d537640 10 monclient: dump: 2024-09-15T04:01:14.229 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.229 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.229 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.230 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.230 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.230 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.230 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.230 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.230 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.230 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d5079d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:01:14.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d4d537640 1 -- 172.21.15.110:0/1236941760 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4000c040 con 0x7f5d30003a00 2024-09-15T04:01:14.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d4d537640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:01:14.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d5079d640 10 monclient: get_monmap_and_config success 2024-09-15T04:01:14.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d4d537640 1 -- 172.21.15.110:0/1236941760 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d40034690 con 0x7f5d30003a00 2024-09-15T04:01:14.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d5079d640 4 set_mon_vals no callback set 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d4d537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d4d537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.215+0000 7f5d4d537640 10 monclient: dump: 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.216+0000 7f5d5079d640 10 monclient: shutdown 2024-09-15T04:01:14.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.216+0000 7f5d5079d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:01:14.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.216+0000 7f5d5079d640 1 -- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d30003a00 msgr2=0x7f5d30003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.216+0000 7f5d5079d640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d30003a00 0x7f5d30003df0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5d40006f60 tx=0x7f5d40008040 comp rx=0 tx=0).stop 2024-09-15T04:01:14.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.216+0000 7f5d5079d640 1 -- 172.21.15.110:0/1236941760 shutdown_connections 2024-09-15T04:01:14.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.216+0000 7f5d5079d640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d30005180 0x7f5d3000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.216+0000 7f5d5079d640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d30004780 0x7f5d30004bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.216+0000 7f5d5079d640 1 --2- 172.21.15.110:0/1236941760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d30003a00 0x7f5d30003df0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.216+0000 7f5d5079d640 1 -- 172.21.15.110:0/1236941760 >> 172.21.15.110:0/1236941760 conn(0x7f5d48063270 msgr2=0x7f5d48063670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:01:14.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.216+0000 7f5d5079d640 1 -- 172.21.15.110:0/1236941760 shutdown_connections 2024-09-15T04:01:14.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.216+0000 7f5d5079d640 1 -- 172.21.15.110:0/1236941760 wait complete. 2024-09-15T04:01:14.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.217+0000 7f5d5079d640 10 monclient: build_initial_monmap 2024-09-15T04:01:14.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.217+0000 7f5d5079d640 10 monclient: monmap: 2024-09-15T04:01:14.232 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:01:14.233 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:01:14.233 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:01:14.233 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:01:14.233 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:01:14.233 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.233 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:01:14.233 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:01:14.233 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:01:14.233 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.217+0000 7f5d5079d640 1 Processor -- start 2024-09-15T04:01:14.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.217+0000 7f5d5079d640 1 -- start start 2024-09-15T04:01:14.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.217+0000 7f5d5079d640 10 monclient: init 2024-09-15T04:01:14.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 10 monclient: _reopen_session rank -1 2024-09-15T04:01:14.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T04:01:14.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d30003a00 0x7f5d4813d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 10 monclient(hunting): picked mon.noname-b con 0x7f5d30003a00 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:01:14.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d30004780 0x7f5d4813c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 10 monclient(hunting): picked mon.noname-c con 0x7f5d30004780 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:01:14.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d30005180 0x7f5d481405c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 10 monclient(hunting): picked mon.noname-a con 0x7f5d30005180 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:01:14.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d48121d90 con 0x7f5d30005180 2024-09-15T04:01:14.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d48121810 con 0x7f5d30003a00 2024-09-15T04:01:14.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4813db60 con 0x7f5d30004780 2024-09-15T04:01:14.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 10 monclient(hunting): _renew_subs 2024-09-15T04:01:14.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d5079d640 10 monclient(hunting): authenticate will time out at 897.862247s 2024-09-15T04:01:14.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4dd38640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d30004780 msgr2=0x7f5d4813c700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:14.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4dd38640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d30004780 0x7f5d4813c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:01:14.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4e539640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d30003a00 0x7f5d4813d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d30005180 0x7f5d481405c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4e539640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d30003a00 0x7f5d4813d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.110:52072/0 (socket says 172.21.15.110:52072) 2024-09-15T04:01:14.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4e539640 1 -- 172.21.15.110:0/1695723782 learned_addr learned my addr 172.21.15.110:0/1695723782 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:01:14.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 10 monclient(hunting): get_auth_request con 0x7f5d30005180 auth_method 0 2024-09-15T04:01:14.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:14.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4e539640 10 monclient(hunting): get_auth_request con 0x7f5d30003a00 auth_method 0 2024-09-15T04:01:14.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4e539640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4e539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:01:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:01:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:01:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4e539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:01:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4e539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:01:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4e539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:01:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 10 monclient(hunting): handle_auth_done global_id 54186 payload 824 2024-09-15T04:01:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1695723782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d30004780 msgr2=0x7f5d4813c700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:01:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1695723782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d30004780 0x7f5d4813c700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1695723782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d30003a00 msgr2=0x7f5d4813d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1695723782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d30003a00 0x7f5d4813d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 10 monclient: _finish_hunting 0 2024-09-15T04:01:14.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 1 monclient: found mon.noname-a 2024-09-15T04:01:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:01:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:01:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.218+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1695723782 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4811fd00 con 0x7f5d30005180 2024-09-15T04:01:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d4ed3a640 1 -- 172.21.15.110:0/1695723782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d30005180 msgr2=0x7f5d481405c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d4ed3a640 1 --2- 172.21.15.110:0/1695723782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d30005180 0x7f5d481405c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5d48068010 tx=0x7f5d3800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d467fc640 1 -- 172.21.15.110:0/1695723782 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d38016840 con 0x7f5d30005180 2024-09-15T04:01:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:01:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d467fc640 10 monclient: dump: 2024-09-15T04:01:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.237 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d467fc640 10 monclient: _finish_auth 0 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d467fc640 1 -- 172.21.15.110:0/1695723782 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3800f050 con 0x7f5d30005180 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d5079d640 5 monclient: authenticate success, global_id 54186 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d5079d640 10 monclient: _renew_subs 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d5079d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d5079d640 1 -- 172.21.15.110:0/1695723782 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d48121ae0 con 0x7f5d30005180 2024-09-15T04:01:14.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d467fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:01:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d467fc640 1 -- 172.21.15.110:0/1695723782 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d38016c10 con 0x7f5d30005180 2024-09-15T04:01:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:01:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d467fc640 10 monclient: dump: 2024-09-15T04:01:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.239 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.239 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d5079d640 10 monclient: _renew_subs 2024-09-15T04:01:14.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d5079d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:01:14.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d5079d640 1 -- 172.21.15.110:0/1695723782 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4813cde0 con 0x7f5d30005180 2024-09-15T04:01:14.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.221+0000 7f5d4cd36640 4 set_mon_vals no callback set 2024-09-15T04:01:14.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.222+0000 7f5d467fc640 1 -- 172.21.15.110:0/1695723782 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f5d38003040 con 0x7f5d30005180 2024-09-15T04:01:14.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.222+0000 7f5d5079d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:01:14.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.222+0000 7f5d5079d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:01:14.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.222+0000 7f5d5079d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:01:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.222+0000 7f5d5079d640 1 -- 172.21.15.110:0/1695723782 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d48121fa0 con 0x7f5d30005180 2024-09-15T04:01:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.222+0000 7f5d467fc640 1 --2- 172.21.15.110:0/1695723782 >> v2:172.21.15.177:6812/28201 conn(0x7f5d34039a50 0x7f5d3403bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.222+0000 7f5d467fc640 1 -- 172.21.15.110:0/1695723782 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f5d3804b520 con 0x7f5d30005180 2024-09-15T04:01:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.222+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1695723782 >> v2:172.21.15.177:6812/28201 conn(0x7f5d34039a50 0x7f5d3403bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.222+0000 7f5d4e539640 10 monclient: get_auth_request con 0x7f5d34039a50 auth_method 0 2024-09-15T04:01:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.225+0000 7f5d4e539640 1 --2- 172.21.15.110:0/1695723782 >> v2:172.21.15.177:6812/28201 conn(0x7f5d34039a50 0x7f5d3403bf00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5d40008080 tx=0x7f5d40037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.227+0000 7f5d467fc640 1 -- 172.21.15.110:0/1695723782 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f5d380215f0 con 0x7f5d30005180 2024-09-15T04:01:14.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.227+0000 7f5d467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:01:14.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.227+0000 7f5d467fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:01:14.289 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:14.289 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30825]: starting. 2024-09-15T04:01:14.289 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30825]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30788 2024-09-15T04:01:14.289 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30825]: created object 0... 2024-09-15T04:01:14.290 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30825]: created object 25... 2024-09-15T04:01:14.290 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30825]: created object 49... 2024-09-15T04:01:14.290 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30825]: finishing. 2024-09-15T04:01:14.290 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30825]: shutting down. 2024-09-15T04:01:14.327 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:14.327 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30827]: starting. 2024-09-15T04:01:14.327 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30827]: listing objects. 2024-09-15T04:01:14.327 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30827]: listed object 0... 2024-09-15T04:01:14.327 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30827]: listed object 25... 2024-09-15T04:01:14.327 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30827]: saw 50 objects 2024-09-15T04:01:14.327 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30827]: shutting down. 2024-09-15T04:01:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.453+0000 7f5d5079d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:01:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.453+0000 7f5d5079d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:01:14.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.453+0000 7f5d5079d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:01:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.453+0000 7f5d5079d640 1 -- 172.21.15.110:0/1695723782 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5d48070950 con 0x7f5d30005180 2024-09-15T04:01:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.454+0000 7f5d467fc640 1 -- 172.21.15.110:0/1695723782 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f5d38021e60 con 0x7f5d30005180 2024-09-15T04:01:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.454+0000 7f5d467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:01:14.455 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.454+0000 7f5d467fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:01:14.455 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":26,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:01:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.455+0000 7f5d5079d640 1 -- 172.21.15.110:0/1695723782 >> v2:172.21.15.177:6812/28201 conn(0x7f5d34039a50 msgr2=0x7f5d3403bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.455+0000 7f5d5079d640 1 --2- 172.21.15.110:0/1695723782 >> v2:172.21.15.177:6812/28201 conn(0x7f5d34039a50 0x7f5d3403bf00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5d40008080 tx=0x7f5d40037000 comp rx=0 tx=0).stop 2024-09-15T04:01:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.455+0000 7f5d5079d640 10 monclient: shutdown 2024-09-15T04:01:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.455+0000 7f5d5079d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:01:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.455+0000 7f5d5079d640 1 -- 172.21.15.110:0/1695723782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d30005180 msgr2=0x7f5d481405c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.455+0000 7f5d5079d640 1 --2- 172.21.15.110:0/1695723782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d30005180 0x7f5d481405c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5d48068010 tx=0x7f5d3800b040 comp rx=0 tx=0).stop 2024-09-15T04:01:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.456+0000 7f5d5079d640 1 -- 172.21.15.110:0/1695723782 shutdown_connections 2024-09-15T04:01:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.456+0000 7f5d5079d640 1 --2- 172.21.15.110:0/1695723782 >> v2:172.21.15.177:6812/28201 conn(0x7f5d34039a50 0x7f5d3403bf00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.456+0000 7f5d5079d640 1 --2- 172.21.15.110:0/1695723782 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5d30005180 0x7f5d481405c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.456+0000 7f5d5079d640 1 --2- 172.21.15.110:0/1695723782 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5d30004780 0x7f5d4813c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.456+0000 7f5d5079d640 1 --2- 172.21.15.110:0/1695723782 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5d30003a00 0x7f5d4813d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.457 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.456+0000 7f5d5079d640 1 -- 172.21.15.110:0/1695723782 >> 172.21.15.110:0/1695723782 conn(0x7f5d48063270 msgr2=0x7f5d480a40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:01:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.456+0000 7f5d5079d640 1 -- 172.21.15.110:0/1695723782 shutdown_connections 2024-09-15T04:01:14.458 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.456+0000 7f5d5079d640 1 -- 172.21.15.110:0/1695723782 wait complete. 2024-09-15T04:01:14.467 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T04:01:14.467 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T04:01:14.467 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T04:01:14.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.559+0000 7f01816a8640 10 monclient: get_monmap_and_config 2024-09-15T04:01:14.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.559+0000 7f01816a8640 10 monclient: build_initial_monmap 2024-09-15T04:01:14.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.559+0000 7f01816a8640 10 monclient: monmap: 2024-09-15T04:01:14.561 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:01:14.561 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.561 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:01:14.560915+0000 2024-09-15T04:01:14.561 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:01:14.560915+0000 2024-09-15T04:01:14.561 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:01:14.561 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.561 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:01:14.561 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:01:14.561 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:01:14.561 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:01:14.561 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:01:14.562 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.560+0000 7f01816a8640 1 Processor -- start 2024-09-15T04:01:14.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.560+0000 7f01816a8640 1 -- start start 2024-09-15T04:01:14.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.560+0000 7f01816a8640 10 monclient: init 2024-09-15T04:01:14.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01816a8640 10 monclient: _reopen_session rank -1 2024-09-15T04:01:14.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01816a8640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-15T04:01:14.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01816a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f017c0a0290 addr v1:172.21.15.177:6789/0 2024-09-15T04:01:14.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01816a8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f017c06a9a0 0x7f017c06ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01816a8640 10 monclient(hunting): picked mon.noname-b con 0x7f017c06a9a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:01:14.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01816a8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f017c1335d0 0x7f017c1359b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01816a8640 10 monclient(hunting): picked mon.noname-a con 0x7f017c1335d0 addr v2:172.21.15.110:3300/0 2024-09-15T04:01:14.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01816a8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017c06ce10 con 0x7f017c0a0290 2024-09-15T04:01:14.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01816a8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01816a8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f017c06c570 con 0x7f017c1335d0 2024-09-15T04:01:14.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01816a8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01816a8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017c06cb30 con 0x7f017c06a9a0 2024-09-15T04:01:14.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01816a8640 10 monclient(hunting): _renew_subs 2024-09-15T04:01:14.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01816a8640 10 monclient(hunting): authenticate will time out at 898.205115s 2024-09-15T04:01:14.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01727fc640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f017c06a9a0 msgr2=0x7f017c06ea50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:14.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f01727fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f017c06a9a0 0x7f017c06ea50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:01:14.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f017c1335d0 0x7f017c1359b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f017c1335d0 0x7f017c1359b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50462/0 (socket says 172.21.15.110:50462) 2024-09-15T04:01:14.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 1 -- 172.21.15.110:0/619338508 learned_addr learned my addr 172.21.15.110:0/619338508 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:01:14.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 10 monclient(hunting): get_auth_request con 0x7f017c1335d0 auth_method 0 2024-09-15T04:01:14.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:14.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:01:14.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:01:14.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:01:14.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 10 monclient(hunting): handle_auth_done global_id 54189 payload 274 2024-09-15T04:01:14.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 1 -- 172.21.15.110:0/619338508 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f017c06a9a0 msgr2=0x7f017c06ea50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:01:14.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 1 --2- 172.21.15.110:0/619338508 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f017c06a9a0 0x7f017c06ea50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 1 -- 172.21.15.110:0/619338508 >> v1:172.21.15.177:6789/0 conn(0x7f017c0a0290 legacy=0x7f017c06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:01:14.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 10 monclient: _finish_hunting 0 2024-09-15T04:01:14.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 1 monclient: found mon.noname-a 2024-09-15T04:01:14.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:01:14.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:01:14.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.561+0000 7f017b7fe640 1 -- 172.21.15.110:0/619338508 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017c0a0660 con 0x7f017c1335d0 2024-09-15T04:01:14.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f017b7fe640 1 -- 172.21.15.110:0/619338508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c1335d0 msgr2=0x7f017c1359b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:14.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f017b7fe640 1 --2- 172.21.15.110:0/619338508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c1335d0 0x7f017c1359b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f017400d1a0 tx=0x7f017400ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe058f2245e405c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:14.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f017a7fc640 1 -- 172.21.15.110:0/619338508 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174009610 con 0x7f017c1335d0 2024-09-15T04:01:14.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f017a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f017a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:01:14.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f017a7fc640 10 monclient: dump: 2024-09-15T04:01:14.574 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.574 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.575 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.575 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.575 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.575 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.575 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.575 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.575 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.575 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f017a7fc640 10 monclient: _finish_auth 0 2024-09-15T04:01:14.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f017a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:01:14.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f017a7fc640 1 -- 172.21.15.110:0/619338508 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0174009770 con 0x7f017c1335d0 2024-09-15T04:01:14.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f01816a8640 5 monclient: authenticate success, global_id 54189 2024-09-15T04:01:14.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f01816a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:01:14.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f017a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:01:14.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f017a7fc640 1 -- 172.21.15.110:0/619338508 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0174009b10 con 0x7f017c1335d0 2024-09-15T04:01:14.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f017a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f017a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:01:14.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f017a7fc640 10 monclient: dump: 2024-09-15T04:01:14.576 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.576 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.576 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.576 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.577 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.577 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.577 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.577 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.577 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.577 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f01816a8640 10 monclient: get_monmap_and_config success 2024-09-15T04:01:14.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f01816a8640 4 set_mon_vals no callback set 2024-09-15T04:01:14.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f01816a8640 10 monclient: shutdown 2024-09-15T04:01:14.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f01816a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:01:14.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f01816a8640 1 -- 172.21.15.110:0/619338508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c1335d0 msgr2=0x7f017c1359b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f01816a8640 1 --2- 172.21.15.110:0/619338508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c1335d0 0x7f017c1359b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f017400d1a0 tx=0x7f017400ba50 comp rx=0 tx=0).stop 2024-09-15T04:01:14.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f01816a8640 1 -- 172.21.15.110:0/619338508 shutdown_connections 2024-09-15T04:01:14.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f01816a8640 1 --2- 172.21.15.110:0/619338508 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c1335d0 0x7f017c1359b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f01816a8640 1 --2- 172.21.15.110:0/619338508 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f017c06a9a0 0x7f017c06ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.564+0000 7f01816a8640 1 -- 172.21.15.110:0/619338508 >> 172.21.15.110:0/619338508 conn(0x7f017c09d680 msgr2=0x7f017c09da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:01:14.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.565+0000 7f01816a8640 1 -- 172.21.15.110:0/619338508 shutdown_connections 2024-09-15T04:01:14.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.565+0000 7f01816a8640 1 -- 172.21.15.110:0/619338508 wait complete. 2024-09-15T04:01:14.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.565+0000 7f01816a8640 10 monclient: build_initial_monmap 2024-09-15T04:01:14.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.565+0000 7f01816a8640 10 monclient: monmap: 2024-09-15T04:01:14.578 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:01:14.578 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:01:14.578 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:01:14.578 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:01:14.578 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:01:14.578 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.579 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:01:14.579 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:01:14.579 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:01:14.579 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.565+0000 7f01816a8640 1 Processor -- start 2024-09-15T04:01:14.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.565+0000 7f01816a8640 1 -- start start 2024-09-15T04:01:14.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.565+0000 7f01816a8640 10 monclient: init 2024-09-15T04:01:14.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 10 monclient: _reopen_session rank -1 2024-09-15T04:01:14.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T04:01:14.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c06a9a0 0x7f017c119f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 10 monclient(hunting): picked mon.noname-a con 0x7f017c06a9a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:01:14.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f017c0a0290 0x7f017c11a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 10 monclient(hunting): picked mon.noname-b con 0x7f017c0a0290 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:01:14.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f017c1335d0 0x7f017c11aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 10 monclient(hunting): picked mon.noname-c con 0x7f017c1335d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:01:14.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c09adf0 con 0x7f017c06a9a0 2024-09-15T04:01:14.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c06e750 con 0x7f017c0a0290 2024-09-15T04:01:14.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017c11d960 con 0x7f017c1335d0 2024-09-15T04:01:14.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 10 monclient(hunting): _renew_subs 2024-09-15T04:01:14.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01816a8640 10 monclient(hunting): authenticate will time out at 898.210400s 2024-09-15T04:01:14.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017b7fe640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f017c1335d0 msgr2=0x7f017c11aa00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:14.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017b7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f017c1335d0 0x7f017c11aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:01:14.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c06a9a0 0x7f017c119f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c06a9a0 0x7f017c119f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50472/0 (socket says 172.21.15.110:50472) 2024-09-15T04:01:14.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 1 -- 172.21.15.110:0/230005289 learned_addr learned my addr 172.21.15.110:0/230005289 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:01:14.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 10 monclient(hunting): get_auth_request con 0x7f017c06a9a0 auth_method 0 2024-09-15T04:01:14.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:14.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:01:14.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:01:14.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:01:14.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01727fc640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f017c0a0290 0x7f017c11a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01727fc640 10 monclient(hunting): get_auth_request con 0x7f017c0a0290 auth_method 0 2024-09-15T04:01:14.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01727fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f01727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:14.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 10 monclient(hunting): handle_auth_done global_id 54192 payload 824 2024-09-15T04:01:14.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 1 -- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f017c1335d0 msgr2=0x7f017c11aa00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:01:14.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f017c1335d0 0x7f017c11aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 1 -- 172.21.15.110:0/230005289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f017c0a0290 msgr2=0x7f017c11a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f017c0a0290 0x7f017c11a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 10 monclient: _finish_hunting 0 2024-09-15T04:01:14.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 1 monclient: found mon.noname-a 2024-09-15T04:01:14.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:01:14.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:01:14.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.566+0000 7f017affd640 1 -- 172.21.15.110:0/230005289 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017400bcf0 con 0x7f017c06a9a0 2024-09-15T04:01:14.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.568+0000 7f017affd640 1 -- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c06a9a0 msgr2=0x7f017c119f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:14.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.568+0000 7f017affd640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c06a9a0 0x7f017c119f80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f016c00c5e0 tx=0x7f016c007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:14.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01737fe640 1 -- 172.21.15.110:0/230005289 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c012840 con 0x7f017c06a9a0 2024-09-15T04:01:14.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:01:14.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01737fe640 10 monclient: dump: 2024-09-15T04:01:14.583 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.583 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.583 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.583 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.583 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.583 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.584 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.584 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.584 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.584 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01737fe640 10 monclient: _finish_auth 0 2024-09-15T04:01:14.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:01:14.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01816a8640 5 monclient: authenticate success, global_id 54192 2024-09-15T04:01:14.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01737fe640 1 -- 172.21.15.110:0/230005289 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016c0129a0 con 0x7f017c06a9a0 2024-09-15T04:01:14.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01816a8640 10 monclient: _renew_subs 2024-09-15T04:01:14.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01816a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:01:14.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01816a8640 1 -- 172.21.15.110:0/230005289 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017c11d3a0 con 0x7f017c06a9a0 2024-09-15T04:01:14.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01737fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:01:14.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01737fe640 1 -- 172.21.15.110:0/230005289 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c012d60 con 0x7f017c06a9a0 2024-09-15T04:01:14.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:01:14.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01737fe640 10 monclient: dump: 2024-09-15T04:01:14.585 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.585 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.585 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.585 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.585 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.585 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.585 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.585 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.585 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.585 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f0179ffb640 4 set_mon_vals no callback set 2024-09-15T04:01:14.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01816a8640 10 monclient: _renew_subs 2024-09-15T04:01:14.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01816a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:01:14.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01816a8640 1 -- 172.21.15.110:0/230005289 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f017c11af40 con 0x7f017c06a9a0 2024-09-15T04:01:14.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01816a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:01:14.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01816a8640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c11c4b0 0x7f017c02aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01816a8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01816a8640 1 -- 172.21.15.110:0/230005289 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c11b1c0 con 0x7f017c11c4b0 2024-09-15T04:01:14.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.569+0000 7f01737fe640 1 -- 172.21.15.110:0/230005289 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f016c011590 con 0x7f017c06a9a0 2024-09-15T04:01:14.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.570+0000 7f01727fc640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c11c4b0 0x7f017c02aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.570+0000 7f01727fc640 10 monclient: get_auth_request con 0x7f017c11c4b0 auth_method 0 2024-09-15T04:01:14.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.570+0000 7f01727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.570+0000 7f01727fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.570+0000 7f01727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:14.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.570+0000 7f01727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:01:14.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.570+0000 7f01727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:01:14.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.570+0000 7f01727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:01:14.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.570+0000 7f01737fe640 1 --2- 172.21.15.110:0/230005289 >> v2:172.21.15.177:6812/28201 conn(0x7f015c038700 0x7f015c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.570+0000 7f01737fe640 1 -- 172.21.15.110:0/230005289 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f016c04b910 con 0x7f017c06a9a0 2024-09-15T04:01:14.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.570+0000 7f01727fc640 10 monclient(hunting): handle_auth_done global_id 54195 payload 824 2024-09-15T04:01:14.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.570+0000 7f01727fc640 1 -- 172.21.15.110:0/230005289 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f017c11be50 con 0x7f017c11c4b0 2024-09-15T04:01:14.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.570+0000 7f017b7fe640 1 --2- 172.21.15.110:0/230005289 >> v2:172.21.15.177:6812/28201 conn(0x7f015c038700 0x7f015c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.570+0000 7f017b7fe640 10 monclient: get_auth_request con 0x7f015c038700 auth_method 0 2024-09-15T04:01:14.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.572+0000 7f01727fc640 1 -- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c11c4b0 msgr2=0x7f017c02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:14.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.572+0000 7f01727fc640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c11c4b0 0x7f017c02aa40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f016400d6f0 tx=0x7f0164004ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:14.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.572+0000 7f01737fe640 1 -- 172.21.15.110:0/230005289 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0164012820 con 0x7f017c11c4b0 2024-09-15T04:01:14.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.572+0000 7f01737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.572+0000 7f01737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:01:14.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.572+0000 7f01737fe640 10 monclient: dump: 2024-09-15T04:01:14.588 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.588 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.588 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.588 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.588 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.588 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.588 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.589 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.589 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.589 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.572+0000 7f01737fe640 1 -- 172.21.15.110:0/230005289 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f0164012980 con 0x7f017c11c4b0 2024-09-15T04:01:14.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.572+0000 7f01737fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:01:14.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.572+0000 7f01737fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:01:14.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.572+0000 7f01737fe640 1 -- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c11c4b0 msgr2=0x7f017c02aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.572+0000 7f01737fe640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c11c4b0 0x7f017c02aa40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f016400d6f0 tx=0x7f0164004ba0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.572+0000 7f01737fe640 1 -- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c11c4b0 msgr2=0x7f017c02aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:01:14.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.572+0000 7f01737fe640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c11c4b0 0x7f017c02aa40 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f016400d6f0 tx=0x7f0164004ba0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.589 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.572+0000 7f017b7fe640 1 --2- 172.21.15.110:0/230005289 >> v2:172.21.15.177:6812/28201 conn(0x7f015c038700 0x7f015c03abb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f017c069600 tx=0x7f0174039000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:14.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.578+0000 7f01816a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:01:14.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.578+0000 7f01816a8640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c05e310 0x7f017c05e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.578+0000 7f01816a8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.578+0000 7f01816a8640 1 -- 172.21.15.110:0/230005289 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c067b00 con 0x7f017c05e310 2024-09-15T04:01:14.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.578+0000 7f01727fc640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c05e310 0x7f017c05e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.578+0000 7f01727fc640 10 monclient: get_auth_request con 0x7f017c05e310 auth_method 0 2024-09-15T04:01:14.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.578+0000 7f01727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.578+0000 7f01727fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.578+0000 7f01727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:14.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.578+0000 7f01727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:01:14.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.578+0000 7f01727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:01:14.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.578+0000 7f01727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:01:14.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.578+0000 7f01727fc640 10 monclient(hunting): handle_auth_done global_id 54198 payload 824 2024-09-15T04:01:14.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.578+0000 7f01727fc640 1 -- 172.21.15.110:0/230005289 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f017c11d5e0 con 0x7f017c05e310 2024-09-15T04:01:14.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01727fc640 1 -- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c05e310 msgr2=0x7f017c05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:14.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01727fc640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c05e310 0x7f017c05e710 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f016400d6f0 tx=0x7f0164018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:14.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01737fe640 1 -- 172.21.15.110:0/230005289 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016401b720 con 0x7f017c05e310 2024-09-15T04:01:14.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:01:14.592 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01737fe640 10 monclient: dump: 2024-09-15T04:01:14.593 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.593 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.593 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.593 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.593 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.593 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.593 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.593 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.593 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.593 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01737fe640 1 -- 172.21.15.110:0/230005289 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7f0164023070 con 0x7f017c05e310 2024-09-15T04:01:14.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01737fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:01:14.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01737fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:01:14.593 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01737fe640 1 -- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c05e310 msgr2=0x7f017c05e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01737fe640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c05e310 0x7f017c05e710 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f016400d6f0 tx=0x7f0164018000 comp rx=0 tx=0).stop 2024-09-15T04:01:14.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01737fe640 1 -- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c05e310 msgr2=0x7f017c05e710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:01:14.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01737fe640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c05e310 0x7f017c05e710 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f016400d6f0 tx=0x7f0164018000 comp rx=0 tx=0).stop 2024-09-15T04:01:14.594 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:01:14.594 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:01:14.595 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:01:14.595 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "leader", 2024-09-15T04:01:14.595 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 26, 2024-09-15T04:01:14.595 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:01:14.595 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:01:14.595 INFO:teuthology.orchestra.run.smithi110.stdout: 1 2024-09-15T04:01:14.596 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:01:14.596 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 0, 2024-09-15T04:01:14.596 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:01:14.596 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:01:14.596 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:01:14.596 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:01:14.596 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:01:14.596 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:01:14.596 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:01:14.596 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:01:14.596 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:01:14.596 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:01:14.596 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:01:14.597 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:01:14.597 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:01:14.597 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:01:14.597 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:01:14.597 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:01:14.597 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:01:14.597 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:01:14.597 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:01:14.597 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:01:14.597 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:01:14.597 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:01:14.597 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:01:14.598 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:01:14.598 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:01:14.598 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.598 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:01:14.598 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:01:14.598 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:01:14.598 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:01:14.598 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:01:14.598 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:01:14.598 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:01:14.598 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:01:14.598 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:01:14.598 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:01:14.599 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:01:14.599 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:01:14.599 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:01:14.599 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:01:14.599 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:01:14.599 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:01:14.599 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:01:14.599 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:01:14.599 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:01:14.599 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:01:14.599 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:01:14.599 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:01:14.599 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:01:14.599 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:01:14.600 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:01:14.600 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:01:14.600 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:01:14.600 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:01:14.600 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.600 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:01:14.600 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.600 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:01:14.600 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:01:14.600 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:01:14.600 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:01:14.600 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.601 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:01:14.601 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:01:14.601 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:01:14.601 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.601 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.601 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:01:14.601 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:01:14.601 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:01:14.601 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.601 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:01:14.601 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.601 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:01:14.601 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:01:14.602 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:01:14.602 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:01:14.602 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:01:14.602 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.602 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.602 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:01:14.602 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:01:14.602 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:01:14.602 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:01:14.602 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.602 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:01:14.602 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:01:14.602 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:01:14.603 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.604 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.604 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:01:14.604 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:01:14.604 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:01:14.604 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:01:14.604 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.604 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:01:14.604 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:01:14.604 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:01:14.604 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.604 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.604 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:01:14.604 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:01:14.604 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:01:14.605 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.605 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:01:14.605 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.605 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:01:14.605 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:01:14.605 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:01:14.605 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:01:14.605 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:01:14.605 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.605 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:01:14.605 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.605 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:01:14.605 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:01:14.605 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.606 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:01:14.606 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:01:14.606 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:01:14.606 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.606 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:01:14.606 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T04:01:14.606 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.606 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:01:14.606 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:01:14.606 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:01:14.606 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.606 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:01:14.606 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:01:14.606 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.607 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:01:14.607 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:01:14.607 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 2 2024-09-15T04:01:14.607 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.607 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:01:14.607 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.607 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:01:14.607 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:01:14.608 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01816a8640 1 -- 172.21.15.110:0/230005289 >> v2:172.21.15.177:6812/28201 conn(0x7f015c038700 msgr2=0x7f015c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01816a8640 1 --2- 172.21.15.110:0/230005289 >> v2:172.21.15.177:6812/28201 conn(0x7f015c038700 0x7f015c03abb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f017c069600 tx=0x7f0174039000 comp rx=0 tx=0).stop 2024-09-15T04:01:14.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01816a8640 10 monclient: shutdown 2024-09-15T04:01:14.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01816a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:01:14.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01816a8640 1 -- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c06a9a0 msgr2=0x7f017c119f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01816a8640 1 --2- 172.21.15.110:0/230005289 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f017c06a9a0 0x7f017c119f80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f016c00c5e0 tx=0x7f016c007ed0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f017b7fe640 1 -- 172.21.15.110:0/230005289 reap_dead start 2024-09-15T04:01:14.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f017b7fe640 1 -- 172.21.15.110:0/230005289 reap_dead start 2024-09-15T04:01:14.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01816a8640 1 -- 172.21.15.110:0/230005289 shutdown_connections 2024-09-15T04:01:14.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01816a8640 1 -- 172.21.15.110:0/230005289 >> 172.21.15.110:0/230005289 conn(0x7f017c09d680 msgr2=0x7f017c09b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:01:14.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01816a8640 1 -- 172.21.15.110:0/230005289 shutdown_connections 2024-09-15T04:01:14.609 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.581+0000 7f01816a8640 1 -- 172.21.15.110:0/230005289 wait complete. 2024-09-15T04:01:14.610 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T04:01:14.610 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T04:01:14.610 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T04:01:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.704+0000 7f8efc24f640 10 monclient: get_monmap_and_config 2024-09-15T04:01:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.704+0000 7f8efc24f640 10 monclient: build_initial_monmap 2024-09-15T04:01:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.704+0000 7f8efc24f640 10 monclient: monmap: 2024-09-15T04:01:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:01:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:01:14.705057+0000 2024-09-15T04:01:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:01:14.705057+0000 2024-09-15T04:01:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:01:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.705 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:01:14.706 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:01:14.706 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:01:14.706 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:01:14.706 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:01:14.706 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.704+0000 7f8efc24f640 1 Processor -- start 2024-09-15T04:01:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.704+0000 7f8efc24f640 1 -- start start 2024-09-15T04:01:14.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.704+0000 7f8efc24f640 10 monclient: init 2024-09-15T04:01:14.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efc24f640 10 monclient: _reopen_session rank -1 2024-09-15T04:01:14.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efc24f640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T04:01:14.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efc24f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ef40a1cf0 addr v1:172.21.15.110:6789/0 2024-09-15T04:01:14.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efc24f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8ef4131a80 0x7f8ef409ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efc24f640 10 monclient(hunting): picked mon.noname-c con 0x7f8ef4131a80 addr v2:172.21.15.177:3300/0 2024-09-15T04:01:14.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efc24f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8ef4136940 0x7f8ef4138d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efc24f640 10 monclient(hunting): picked mon.noname-a con 0x7f8ef4136940 addr v2:172.21.15.110:3300/0 2024-09-15T04:01:14.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efc24f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef409cb70 con 0x7f8ef40a1cf0 2024-09-15T04:01:14.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efc24f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efc24f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef40a20c0 con 0x7f8ef4136940 2024-09-15T04:01:14.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efc24f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efc24f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef409c330 con 0x7f8ef4131a80 2024-09-15T04:01:14.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efc24f640 10 monclient(hunting): _renew_subs 2024-09-15T04:01:14.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efc24f640 10 monclient(hunting): authenticate will time out at 898.349334s 2024-09-15T04:01:14.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8ef4136940 0x7f8ef4138d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8ef9feb640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8ef40a1cf0 0x7f8ef409e660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:34768/0 (socket says 172.21.15.110:34768) 2024-09-15T04:01:14.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8ef9feb640 1 -- 172.21.15.110:0/2061102239 learned_addr learned my addr 172.21.15.110:0/2061102239 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:01:14.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 1 --2- 172.21.15.110:0/2061102239 >> v2:172.21.15.110:3300/0 conn(0x7f8ef4136940 0x7f8ef4138d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50488/0 (socket says 172.21.15.110:50488) 2024-09-15T04:01:14.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 10 monclient(hunting): get_auth_request con 0x7f8ef4136940 auth_method 0 2024-09-15T04:01:14.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:14.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:01:14.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:01:14.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:01:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8ef97ea640 1 --2- 172.21.15.110:0/2061102239 >> v2:172.21.15.177:3300/0 conn(0x7f8ef4131a80 0x7f8ef409ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 10 monclient(hunting): handle_auth_done global_id 54201 payload 274 2024-09-15T04:01:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 1 -- 172.21.15.110:0/2061102239 >> v2:172.21.15.177:3300/0 conn(0x7f8ef4131a80 msgr2=0x7f8ef409ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:01:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 1 --2- 172.21.15.110:0/2061102239 >> v2:172.21.15.177:3300/0 conn(0x7f8ef4131a80 0x7f8ef409ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 1 -- 172.21.15.110:0/2061102239 >> v1:172.21.15.110:6789/0 conn(0x7f8ef40a1cf0 legacy=0x7f8ef409e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:01:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 10 monclient: _finish_hunting 0 2024-09-15T04:01:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 1 monclient: found mon.noname-a 2024-09-15T04:01:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:01:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:01:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8efa7ec640 1 -- 172.21.15.110:0/2061102239 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef4131fb0 con 0x7f8ef4136940 2024-09-15T04:01:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8ef97ea640 10 monclient: get_auth_request con 0x7f8ef4131a80 auth_method 0 2024-09-15T04:01:14.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.705+0000 7f8ef97ea640 1 --2- 172.21.15.110:0/2061102239 >> v2:172.21.15.177:3300/0 conn(0x7f8ef4131a80 0x7f8ef409ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:01:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8efa7ec640 1 -- 172.21.15.110:0/2061102239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef4136940 msgr2=0x7f8ef4138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8efa7ec640 1 --2- 172.21.15.110:0/2061102239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef4136940 0x7f8ef4138d20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4068dc0 tx=0x7f8eec009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e5ba9812e3ca4ba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8ef8fe9640 1 -- 172.21.15.110:0/2061102239 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec007840 con 0x7f8ef4136940 2024-09-15T04:01:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8ef8fe9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8ef8fe9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:01:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8ef8fe9640 10 monclient: dump: 2024-09-15T04:01:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.719 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.720 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8ef8fe9640 10 monclient: _finish_auth 0 2024-09-15T04:01:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8ef8fe9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:01:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8efc24f640 5 monclient: authenticate success, global_id 54201 2024-09-15T04:01:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8efc24f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:01:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8ef8fe9640 1 -- 172.21.15.110:0/2061102239 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec02d050 con 0x7f8ef4136940 2024-09-15T04:01:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8ef8fe9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:01:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8efc24f640 10 monclient: get_monmap_and_config success 2024-09-15T04:01:14.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8efc24f640 4 set_mon_vals no callback set 2024-09-15T04:01:14.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8ef8fe9640 1 -- 172.21.15.110:0/2061102239 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec007c10 con 0x7f8ef4136940 2024-09-15T04:01:14.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8ef8fe9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8ef8fe9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:01:14.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.708+0000 7f8ef8fe9640 10 monclient: dump: 2024-09-15T04:01:14.721 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.721 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.721 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.721 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.721 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.721 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.721 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.721 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.721 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.722 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.709+0000 7f8efc24f640 10 monclient: shutdown 2024-09-15T04:01:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.709+0000 7f8efc24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:01:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.709+0000 7f8efc24f640 1 -- 172.21.15.110:0/2061102239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef4136940 msgr2=0x7f8ef4138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.709+0000 7f8efc24f640 1 --2- 172.21.15.110:0/2061102239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef4136940 0x7f8ef4138d20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4068dc0 tx=0x7f8eec009780 comp rx=0 tx=0).stop 2024-09-15T04:01:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.709+0000 7f8efc24f640 1 -- 172.21.15.110:0/2061102239 shutdown_connections 2024-09-15T04:01:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.709+0000 7f8efc24f640 1 --2- 172.21.15.110:0/2061102239 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef4136940 0x7f8ef4138d20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.709+0000 7f8efc24f640 1 --2- 172.21.15.110:0/2061102239 >> v2:172.21.15.177:3300/0 conn(0x7f8ef4131a80 0x7f8ef409ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.709+0000 7f8efc24f640 1 -- 172.21.15.110:0/2061102239 >> 172.21.15.110:0/2061102239 conn(0x7f8ef4062f70 msgr2=0x7f8ef4063340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:01:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.709+0000 7f8efc24f640 1 -- 172.21.15.110:0/2061102239 shutdown_connections 2024-09-15T04:01:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.709+0000 7f8efc24f640 1 -- 172.21.15.110:0/2061102239 wait complete. 2024-09-15T04:01:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.709+0000 7f8efc24f640 10 monclient: build_initial_monmap 2024-09-15T04:01:14.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.709+0000 7f8efc24f640 10 monclient: monmap: 2024-09-15T04:01:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:01:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:01:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:01:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:01:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:01:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:01:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:01:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:01:14.723 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 1 Processor -- start 2024-09-15T04:01:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 1 -- start start 2024-09-15T04:01:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 10 monclient: init 2024-09-15T04:01:14.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 10 monclient: _reopen_session rank -1 2024-09-15T04:01:14.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T04:01:14.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef40a1cf0 0x7f8ef40996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 10 monclient(hunting): picked mon.noname-c con 0x7f8ef40a1cf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:01:14.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef4131a80 0x7f8ef4099bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 10 monclient(hunting): picked mon.noname-a con 0x7f8ef4131a80 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:01:14.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef4136940 0x7f8ef409a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 10 monclient(hunting): picked mon.noname-b con 0x7f8ef4136940 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:01:14.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef411c6c0 con 0x7f8ef4131a80 2024-09-15T04:01:14.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8ef9feb640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef40a1cf0 msgr2=0x7f8ef40996b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:01:14.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef40a5680 con 0x7f8ef4136940 2024-09-15T04:01:14.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef409afb0 con 0x7f8ef40a1cf0 2024-09-15T04:01:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8ef9feb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef40a1cf0 0x7f8ef40996b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:01:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 10 monclient(hunting): _renew_subs 2024-09-15T04:01:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efc24f640 10 monclient(hunting): authenticate will time out at 898.354735s 2024-09-15T04:01:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8ef97ea640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef4131a80 0x7f8ef4099bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8ef97ea640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef4131a80 0x7f8ef4099bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:50504/0 (socket says 172.21.15.110:50504) 2024-09-15T04:01:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8ef97ea640 1 -- 172.21.15.110:0/1942523488 learned_addr learned my addr 172.21.15.110:0/1942523488 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:01:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8ef97ea640 10 monclient(hunting): get_auth_request con 0x7f8ef4131a80 auth_method 0 2024-09-15T04:01:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8ef97ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8ef97ea640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8ef97ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8ef97ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:01:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8ef97ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:01:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8ef97ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:01:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efa7ec640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef4136940 0x7f8ef409a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efa7ec640 10 monclient(hunting): get_auth_request con 0x7f8ef4136940 auth_method 0 2024-09-15T04:01:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efa7ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efa7ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.710+0000 7f8efa7ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.711+0000 7f8ef97ea640 10 monclient(hunting): handle_auth_done global_id 54204 payload 824 2024-09-15T04:01:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.711+0000 7f8ef97ea640 1 -- 172.21.15.110:0/1942523488 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef40a1cf0 msgr2=0x7f8ef40996b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:01:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.711+0000 7f8ef97ea640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8ef40a1cf0 0x7f8ef40996b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.711+0000 7f8ef97ea640 1 -- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef4136940 msgr2=0x7f8ef409a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.711+0000 7f8ef97ea640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef4136940 0x7f8ef409a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:01:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.711+0000 7f8ef97ea640 10 monclient: _finish_hunting 0 2024-09-15T04:01:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.711+0000 7f8ef97ea640 1 monclient: found mon.noname-a 2024-09-15T04:01:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.711+0000 7f8ef97ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:01:14.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.711+0000 7f8ef97ea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:01:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.711+0000 7f8ef97ea640 1 -- 172.21.15.110:0/1942523488 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec009a20 con 0x7f8ef4131a80 2024-09-15T04:01:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8ef97ea640 1 -- 172.21.15.110:0/1942523488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef4131a80 msgr2=0x7f8ef4099bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8ef97ea640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef4131a80 0x7f8ef4099bf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8ee400c8b0 tx=0x7f8ee4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/1942523488 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee400ce40 con 0x7f8ef4131a80 2024-09-15T04:01:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8ee9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8ee9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:01:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8ee9ffb640 10 monclient: dump: 2024-09-15T04:01:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.727 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.728 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8ee9ffb640 10 monclient: _finish_auth 0 2024-09-15T04:01:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8ee9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:01:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/1942523488 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee4007aa0 con 0x7f8ef4131a80 2024-09-15T04:01:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8efc24f640 5 monclient: authenticate success, global_id 54204 2024-09-15T04:01:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8efc24f640 10 monclient: _renew_subs 2024-09-15T04:01:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8efc24f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:01:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8efc24f640 1 -- 172.21.15.110:0/1942523488 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef411c3e0 con 0x7f8ef4131a80 2024-09-15T04:01:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8ee9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:01:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/1942523488 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee4007e60 con 0x7f8ef4131a80 2024-09-15T04:01:14.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8ee9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8ee9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:01:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8ee9ffb640 10 monclient: dump: 2024-09-15T04:01:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.729 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8eebfff640 4 set_mon_vals no callback set 2024-09-15T04:01:14.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8efc24f640 10 monclient: _renew_subs 2024-09-15T04:01:14.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8efc24f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:01:14.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.713+0000 7f8efc24f640 1 -- 172.21.15.110:0/1942523488 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef411b480 con 0x7f8ef4131a80 2024-09-15T04:01:14.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/1942523488 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f8ee4011600 con 0x7f8ef4131a80 2024-09-15T04:01:14.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8efc24f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:01:14.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8efc24f640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef409a670 0x7f8ef4095a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8efc24f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8efc24f640 1 -- 172.21.15.110:0/1942523488 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef406f1e0 con 0x7f8ef409a670 2024-09-15T04:01:14.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8ee9ffb640 1 --2- 172.21.15.110:0/1942523488 >> v2:172.21.15.177:6812/28201 conn(0x7f8ed8038700 0x7f8ed803abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/1942523488 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f8ee404b850 con 0x7f8ef4131a80 2024-09-15T04:01:14.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8ef9feb640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef409a670 0x7f8ef4095a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8ef9feb640 10 monclient: get_auth_request con 0x7f8ef409a670 auth_method 0 2024-09-15T04:01:14.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8ef9feb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8ef9feb640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8ef9feb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:14.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8efa7ec640 1 --2- 172.21.15.110:0/1942523488 >> v2:172.21.15.177:6812/28201 conn(0x7f8ed8038700 0x7f8ed803abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8efa7ec640 10 monclient: get_auth_request con 0x7f8ed8038700 auth_method 0 2024-09-15T04:01:14.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8ef9feb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:01:14.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8ef9feb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:01:14.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.714+0000 7f8ef9feb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:01:14.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.715+0000 7f8ef9feb640 10 monclient(hunting): handle_auth_done global_id 4846 payload 824 2024-09-15T04:01:14.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.715+0000 7f8ef9feb640 1 -- 172.21.15.110:0/1942523488 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8ef40647e0 con 0x7f8ef409a670 2024-09-15T04:01:14.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.717+0000 7f8ef9feb640 1 -- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef409a670 msgr2=0x7f8ef4095a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:14.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.717+0000 7f8ef9feb640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef409a670 0x7f8ef4095a70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8ef00047d0 tx=0x7f8ef0007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:14.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.717+0000 7f8efa7ec640 1 --2- 172.21.15.110:0/1942523488 >> v2:172.21.15.177:6812/28201 conn(0x7f8ed8038700 0x7f8ed803abb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8eec034000 tx=0x7f8eec037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:14.731 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.717+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/1942523488 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0005d70 con 0x7f8ef409a670 2024-09-15T04:01:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.717+0000 7f8ee9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.717+0000 7f8ee9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:01:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.717+0000 7f8ee9ffb640 10 monclient: dump: 2024-09-15T04:01:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.732 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.732 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.717+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/1942523488 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8ef00070d0 con 0x7f8ef409a670 2024-09-15T04:01:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.717+0000 7f8ee9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:01:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.717+0000 7f8ee9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:01:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.717+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef409a670 msgr2=0x7f8ef4095a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.717+0000 7f8ee9ffb640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef409a670 0x7f8ef4095a70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8ef00047d0 tx=0x7f8ef0007f00 comp rx=0 tx=0).stop 2024-09-15T04:01:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.717+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef409a670 msgr2=0x7f8ef4095a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:01:14.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.717+0000 7f8ee9ffb640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef409a670 0x7f8ef4095a70 secure :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8ef00047d0 tx=0x7f8ef0007f00 comp rx=0 tx=0).stop 2024-09-15T04:01:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.723+0000 7f8efc24f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:01:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.723+0000 7f8efc24f640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef4095fb0 0x7f8ef40963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:01:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.723+0000 7f8efc24f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:01:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.723+0000 7f8efc24f640 1 -- 172.21.15.110:0/1942523488 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef411ca60 con 0x7f8ef4095fb0 2024-09-15T04:01:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.723+0000 7f8ef9feb640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef4095fb0 0x7f8ef40963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:01:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.723+0000 7f8ef9feb640 10 monclient: get_auth_request con 0x7f8ef4095fb0 auth_method 0 2024-09-15T04:01:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.723+0000 7f8ef9feb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:01:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.723+0000 7f8ef9feb640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:01:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.723+0000 7f8ef9feb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:01:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.723+0000 7f8ef9feb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:01:14.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.723+0000 7f8ef9feb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:01:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.723+0000 7f8ef9feb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:01:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.724+0000 7f8ef9feb640 10 monclient(hunting): handle_auth_done global_id 4849 payload 824 2024-09-15T04:01:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.724+0000 7f8ef9feb640 1 -- 172.21.15.110:0/1942523488 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8ef4067130 con 0x7f8ef4095fb0 2024-09-15T04:01:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.726+0000 7f8ef9feb640 1 -- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef4095fb0 msgr2=0x7f8ef40963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:01:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.726+0000 7f8ef9feb640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef4095fb0 0x7f8ef40963b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0007fd0 tx=0x7f8ef0012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:01:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.726+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/1942523488 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef001f610 con 0x7f8ef4095fb0 2024-09-15T04:01:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.726+0000 7f8ee9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:01:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.726+0000 7f8ee9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:01:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.726+0000 7f8ee9ffb640 10 monclient: dump: 2024-09-15T04:01:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:01:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:01:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.736 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:01:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:01:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:01:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:01:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:01:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:01:14.737 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:01:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.726+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/1942523488 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f8ef001f770 con 0x7f8ef4095fb0 2024-09-15T04:01:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.726+0000 7f8ee9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:01:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.726+0000 7f8ee9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:01:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.726+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef4095fb0 msgr2=0x7f8ef40963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.726+0000 7f8ee9ffb640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef4095fb0 0x7f8ef40963b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0007fd0 tx=0x7f8ef0012000 comp rx=0 tx=0).stop 2024-09-15T04:01:14.737 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.726+0000 7f8ee9ffb640 1 -- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef4095fb0 msgr2=0x7f8ef40963b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:01:14.738 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.726+0000 7f8ee9ffb640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8ef4095fb0 0x7f8ef40963b0 secure :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0007fd0 tx=0x7f8ef0012000 comp rx=0 tx=0).stop 2024-09-15T04:01:14.738 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:01:14.738 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:01:14.738 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:01:14.738 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T04:01:14.738 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 26, 2024-09-15T04:01:14.738 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:01:14.738 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:01:14.738 INFO:teuthology.orchestra.run.smithi110.stdout: 1 2024-09-15T04:01:14.739 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:01:14.739 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 0, 2024-09-15T04:01:14.739 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:01:14.739 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:01:14.739 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:01:14.739 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:01:14.739 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:01:14.739 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:01:14.739 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:01:14.739 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:01:14.739 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:01:14.739 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:01:14.739 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:01:14.739 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:01:14.740 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:01:14.740 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:01:14.740 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:01:14.740 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:01:14.740 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:01:14.740 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:01:14.740 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:01:14.740 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:01:14.740 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:01:14.740 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:01:14.740 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:01:14.740 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:01:14.740 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:01:14.741 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.741 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:01:14.741 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:01:14.741 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:01:14.741 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:01:14.741 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:01:14.741 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:01:14.741 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:01:14.741 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:01:14.741 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:01:14.742 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:01:14.742 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:01:14.742 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:01:14.742 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:01:14.742 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:01:14.742 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:01:14.742 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:01:14.742 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:01:14.742 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:01:14.742 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:01:14.742 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:01:14.743 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:01:14.743 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:01:14.743 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:01:14.743 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:01:14.743 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:01:14.743 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:01:14.743 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:01:14.743 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:01:14.744 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.744 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:01:14.744 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.744 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:01:14.744 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:01:14.744 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:01:14.744 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:01:14.744 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.744 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:01:14.744 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:01:14.744 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:01:14.744 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.744 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.745 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:01:14.745 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:01:14.745 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:01:14.745 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.745 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:01:14.745 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.745 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:01:14.745 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:01:14.745 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:01:14.745 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:01:14.745 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:01:14.745 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.745 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.746 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:01:14.746 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:01:14.746 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:01:14.746 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:01:14.746 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.746 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:01:14.746 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:01:14.746 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:01:14.746 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.746 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.746 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:01:14.746 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:01:14.746 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:01:14.747 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.747 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:01:14.747 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.747 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:01:14.747 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:01:14.747 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:01:14.747 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:01:14.747 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:01:14.747 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.748 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.748 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:01:14.748 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:01:14.748 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:01:14.748 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:01:14.748 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.748 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:01:14.748 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:01:14.748 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:01:14.748 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.748 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.748 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:01:14.748 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:01:14.749 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:01:14.749 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.749 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:01:14.749 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.749 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:01:14.749 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:01:14.749 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:01:14.749 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:01:14.749 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:01:14.749 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.749 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:01:14.749 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.749 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:01:14.750 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:01:14.750 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.750 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:01:14.750 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:01:14.750 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:01:14.750 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.750 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:01:14.750 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T04:01:14.750 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.750 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:01:14.750 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:01:14.750 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 5 2024-09-15T04:01:14.750 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.751 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:01:14.751 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:01:14.751 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.751 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:01:14.751 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:01:14.751 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 6 2024-09-15T04:01:14.751 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.751 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:01:14.751 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": [ 2024-09-15T04:01:14.751 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:01:14.751 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:01:14.751 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:01:14.751 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:01:14.751 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:01:14.752 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:01:14.752 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:01:14.752 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:01:14.752 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:01:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.727+0000 7f8efc24f640 1 -- 172.21.15.110:0/1942523488 >> v2:172.21.15.177:6812/28201 conn(0x7f8ed8038700 msgr2=0x7f8ed803abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.727+0000 7f8efc24f640 1 --2- 172.21.15.110:0/1942523488 >> v2:172.21.15.177:6812/28201 conn(0x7f8ed8038700 0x7f8ed803abb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8eec034000 tx=0x7f8eec037000 comp rx=0 tx=0).stop 2024-09-15T04:01:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.727+0000 7f8efc24f640 10 monclient: shutdown 2024-09-15T04:01:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.727+0000 7f8efc24f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:01:14.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.727+0000 7f8efc24f640 1 -- 172.21.15.110:0/1942523488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef4131a80 msgr2=0x7f8ef4099bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:01:14.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.727+0000 7f8efc24f640 1 --2- 172.21.15.110:0/1942523488 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ef4131a80 0x7f8ef4099bf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8ee400c8b0 tx=0x7f8ee4008040 comp rx=0 tx=0).stop 2024-09-15T04:01:14.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.727+0000 7f8efa7ec640 1 -- 172.21.15.110:0/1942523488 reap_dead start 2024-09-15T04:01:14.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.727+0000 7f8efa7ec640 1 -- 172.21.15.110:0/1942523488 reap_dead start 2024-09-15T04:01:14.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.727+0000 7f8efc24f640 1 -- 172.21.15.110:0/1942523488 shutdown_connections 2024-09-15T04:01:14.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.727+0000 7f8efc24f640 1 -- 172.21.15.110:0/1942523488 >> 172.21.15.110:0/1942523488 conn(0x7f8ef4062f70 msgr2=0x7f8ef40a2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:01:14.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.727+0000 7f8efc24f640 1 -- 172.21.15.110:0/1942523488 shutdown_connections 2024-09-15T04:01:14.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:01:14.727+0000 7f8efc24f640 1 -- 172.21.15.110:0/1942523488 wait complete. 2024-09-15T04:01:14.754 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T04:01:14.754 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T04:01:15.203 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:15.203 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30826]: starting. 2024-09-15T04:01:15.203 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30826]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30788 2024-09-15T04:01:15.204 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30826]: shutting down. 2024-09-15T04:01:15.204 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:15.204 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:15.204 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:01:15.207 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:01:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30909]: starting. 2024-09-15T04:01:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30909]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30908 2024-09-15T04:01:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30909]: created object 0... 2024-09-15T04:01:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30909]: created object 25... 2024-09-15T04:01:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30909]: created object 49... 2024-09-15T04:01:17.320 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30909]: finishing. 2024-09-15T04:01:17.321 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[30909]: shutting down. 2024-09-15T04:01:18.211 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30910]: starting. 2024-09-15T04:01:18.211 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30910]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30908 2024-09-15T04:01:18.211 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[30910]: shutting down. 2024-09-15T04:01:20.346 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:20.346 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30943]: starting. 2024-09-15T04:01:20.346 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30943]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-30908 2024-09-15T04:01:20.346 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30943]: created object 0... 2024-09-15T04:01:20.347 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30943]: created object 25... 2024-09-15T04:01:20.347 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30943]: created object 49... 2024-09-15T04:01:20.347 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30943]: finishing. 2024-09-15T04:01:20.347 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[30943]: shutting down. 2024-09-15T04:01:20.383 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:20.383 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30945]: starting. 2024-09-15T04:01:20.383 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30945]: listing objects. 2024-09-15T04:01:20.383 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30945]: listed object 0... 2024-09-15T04:01:20.384 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30945]: listed object 25... 2024-09-15T04:01:20.384 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30945]: saw 50 objects 2024-09-15T04:01:20.384 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[30945]: shutting down. 2024-09-15T04:01:21.232 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:21.232 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30944]: starting. 2024-09-15T04:01:21.232 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30944]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-30908 2024-09-15T04:01:21.232 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[30944]: shutting down. 2024-09-15T04:01:21.233 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:21.233 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:21.233 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:01:21.235 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:01:22.866 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31012]: starting. 2024-09-15T04:01:22.866 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31012]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31011 2024-09-15T04:01:22.866 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31012]: created object 0... 2024-09-15T04:01:22.866 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31012]: created object 25... 2024-09-15T04:01:22.866 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31012]: created object 49... 2024-09-15T04:01:22.866 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31012]: finishing. 2024-09-15T04:01:22.867 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31012]: shutting down. 2024-09-15T04:01:23.775 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31013]: starting. 2024-09-15T04:01:23.775 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31013]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31011 2024-09-15T04:01:23.775 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31013]: shutting down. 2024-09-15T04:01:25.898 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:25.898 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31046]: starting. 2024-09-15T04:01:25.898 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31046]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31011 2024-09-15T04:01:25.898 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31046]: created object 0... 2024-09-15T04:01:25.898 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31046]: created object 25... 2024-09-15T04:01:25.899 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31046]: created object 49... 2024-09-15T04:01:25.899 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31046]: finishing. 2024-09-15T04:01:25.899 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31046]: shutting down. 2024-09-15T04:01:25.936 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:25.936 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31048]: starting. 2024-09-15T04:01:25.936 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31048]: listing objects. 2024-09-15T04:01:25.936 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31048]: listed object 0... 2024-09-15T04:01:25.936 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31048]: listed object 25... 2024-09-15T04:01:25.936 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31048]: saw 50 objects 2024-09-15T04:01:25.936 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31048]: shutting down. 2024-09-15T04:01:26.791 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:26.791 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31047]: starting. 2024-09-15T04:01:26.792 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31047]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31011 2024-09-15T04:01:26.792 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31047]: shutting down. 2024-09-15T04:01:26.792 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:26.792 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:26.792 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:01:26.795 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:01:28.879 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31115]: starting. 2024-09-15T04:01:28.879 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31115]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31114 2024-09-15T04:01:28.879 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31115]: created object 0... 2024-09-15T04:01:28.879 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31115]: created object 25... 2024-09-15T04:01:28.879 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31115]: created object 49... 2024-09-15T04:01:28.880 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31115]: finishing. 2024-09-15T04:01:28.880 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31115]: shutting down. 2024-09-15T04:01:29.802 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31116]: starting. 2024-09-15T04:01:29.802 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31116]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31114 2024-09-15T04:01:29.802 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31116]: shutting down. 2024-09-15T04:01:31.867 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:31.868 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31149]: starting. 2024-09-15T04:01:31.868 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31149]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31114 2024-09-15T04:01:31.868 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31149]: created object 0... 2024-09-15T04:01:31.868 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31149]: created object 25... 2024-09-15T04:01:31.868 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31149]: created object 49... 2024-09-15T04:01:31.868 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31149]: finishing. 2024-09-15T04:01:31.868 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31149]: shutting down. 2024-09-15T04:01:31.905 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:31.905 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31151]: starting. 2024-09-15T04:01:31.906 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31151]: listing objects. 2024-09-15T04:01:31.906 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31151]: listed object 0... 2024-09-15T04:01:31.906 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31151]: listed object 25... 2024-09-15T04:01:31.906 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31151]: saw 50 objects 2024-09-15T04:01:31.906 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31151]: shutting down. 2024-09-15T04:01:32.777 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:32.777 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31150]: starting. 2024-09-15T04:01:32.777 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31150]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31114 2024-09-15T04:01:32.777 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31150]: shutting down. 2024-09-15T04:01:32.778 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:32.778 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:32.778 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:01:32.780 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:01:34.907 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31218]: starting. 2024-09-15T04:01:34.907 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31218]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31217 2024-09-15T04:01:34.907 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31218]: created object 0... 2024-09-15T04:01:34.908 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31218]: created object 25... 2024-09-15T04:01:34.908 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31218]: created object 49... 2024-09-15T04:01:34.908 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31218]: finishing. 2024-09-15T04:01:34.908 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31218]: shutting down. 2024-09-15T04:01:35.823 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31219]: starting. 2024-09-15T04:01:35.823 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31219]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31217 2024-09-15T04:01:35.824 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31219]: shutting down. 2024-09-15T04:01:37.935 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:37.935 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31252]: starting. 2024-09-15T04:01:37.936 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31252]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31217 2024-09-15T04:01:37.936 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31252]: created object 0... 2024-09-15T04:01:37.936 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31252]: created object 25... 2024-09-15T04:01:37.936 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31252]: created object 49... 2024-09-15T04:01:37.936 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31252]: finishing. 2024-09-15T04:01:37.936 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31252]: shutting down. 2024-09-15T04:01:37.974 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:37.974 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31254]: starting. 2024-09-15T04:01:37.974 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31254]: listing objects. 2024-09-15T04:01:37.974 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31254]: listed object 0... 2024-09-15T04:01:37.974 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31254]: listed object 25... 2024-09-15T04:01:37.974 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31254]: saw 50 objects 2024-09-15T04:01:37.974 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31254]: shutting down. 2024-09-15T04:01:38.837 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:38.837 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31253]: starting. 2024-09-15T04:01:38.837 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31253]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31217 2024-09-15T04:01:38.837 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31253]: shutting down. 2024-09-15T04:01:38.838 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:38.838 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:38.838 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:01:38.841 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:01:40.937 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31321]: starting. 2024-09-15T04:01:40.938 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31321]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31320 2024-09-15T04:01:40.938 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31321]: created object 0... 2024-09-15T04:01:40.938 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31321]: created object 25... 2024-09-15T04:01:40.938 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31321]: created object 49... 2024-09-15T04:01:40.938 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31321]: finishing. 2024-09-15T04:01:40.938 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31321]: shutting down. 2024-09-15T04:01:41.779 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31322]: starting. 2024-09-15T04:01:41.780 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31322]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31320 2024-09-15T04:01:41.780 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31322]: shutting down. 2024-09-15T04:01:43.871 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:43.872 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31355]: starting. 2024-09-15T04:01:43.872 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31355]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31320 2024-09-15T04:01:43.872 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31355]: created object 0... 2024-09-15T04:01:43.872 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31355]: created object 25... 2024-09-15T04:01:43.872 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31355]: created object 49... 2024-09-15T04:01:43.872 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31355]: finishing. 2024-09-15T04:01:43.872 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31355]: shutting down. 2024-09-15T04:01:43.910 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:43.910 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31357]: starting. 2024-09-15T04:01:43.910 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31357]: listing objects. 2024-09-15T04:01:43.910 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31357]: listed object 0... 2024-09-15T04:01:43.911 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31357]: listed object 25... 2024-09-15T04:01:43.911 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31357]: saw 50 objects 2024-09-15T04:01:43.911 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31357]: shutting down. 2024-09-15T04:01:44.786 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:44.786 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31356]: starting. 2024-09-15T04:01:44.786 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31356]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31320 2024-09-15T04:01:44.787 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31356]: shutting down. 2024-09-15T04:01:44.787 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:44.787 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:44.788 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:01:44.790 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:01:46.871 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31424]: starting. 2024-09-15T04:01:46.872 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31424]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31423 2024-09-15T04:01:46.872 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31424]: created object 0... 2024-09-15T04:01:46.872 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31424]: created object 25... 2024-09-15T04:01:46.872 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31424]: created object 49... 2024-09-15T04:01:46.872 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31424]: finishing. 2024-09-15T04:01:46.872 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31424]: shutting down. 2024-09-15T04:01:47.791 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31425]: starting. 2024-09-15T04:01:47.791 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31425]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31423 2024-09-15T04:01:47.791 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31425]: shutting down. 2024-09-15T04:01:49.890 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:49.891 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31458]: starting. 2024-09-15T04:01:49.891 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31458]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31423 2024-09-15T04:01:49.891 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31458]: created object 0... 2024-09-15T04:01:49.891 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31458]: created object 25... 2024-09-15T04:01:49.891 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31458]: created object 49... 2024-09-15T04:01:49.891 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31458]: finishing. 2024-09-15T04:01:49.891 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31458]: shutting down. 2024-09-15T04:01:49.929 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:49.929 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31460]: starting. 2024-09-15T04:01:49.929 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31460]: listing objects. 2024-09-15T04:01:49.929 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31460]: listed object 0... 2024-09-15T04:01:49.929 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31460]: listed object 25... 2024-09-15T04:01:49.929 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31460]: saw 50 objects 2024-09-15T04:01:49.929 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31460]: shutting down. 2024-09-15T04:01:50.805 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:50.806 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31459]: starting. 2024-09-15T04:01:50.806 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31459]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31423 2024-09-15T04:01:50.806 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31459]: shutting down. 2024-09-15T04:01:50.806 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:50.806 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:50.806 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:01:50.809 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:01:52.859 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31527]: starting. 2024-09-15T04:01:52.859 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31527]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31526 2024-09-15T04:01:52.859 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31527]: created object 0... 2024-09-15T04:01:52.859 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31527]: created object 25... 2024-09-15T04:01:52.859 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31527]: created object 49... 2024-09-15T04:01:52.859 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31527]: finishing. 2024-09-15T04:01:52.859 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31527]: shutting down. 2024-09-15T04:01:53.787 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31528]: starting. 2024-09-15T04:01:53.787 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31528]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31526 2024-09-15T04:01:53.787 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31528]: shutting down. 2024-09-15T04:01:55.900 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:55.900 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31561]: starting. 2024-09-15T04:01:55.900 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31561]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31526 2024-09-15T04:01:55.900 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31561]: created object 0... 2024-09-15T04:01:55.900 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31561]: created object 25... 2024-09-15T04:01:55.900 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31561]: created object 49... 2024-09-15T04:01:55.900 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31561]: finishing. 2024-09-15T04:01:55.900 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31561]: shutting down. 2024-09-15T04:01:55.937 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:55.937 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31563]: starting. 2024-09-15T04:01:55.937 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31563]: listing objects. 2024-09-15T04:01:55.937 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31563]: listed object 0... 2024-09-15T04:01:55.937 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31563]: listed object 25... 2024-09-15T04:01:55.937 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31563]: saw 50 objects 2024-09-15T04:01:55.938 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31563]: shutting down. 2024-09-15T04:01:56.810 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:56.810 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31562]: starting. 2024-09-15T04:01:56.810 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31562]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31526 2024-09-15T04:01:56.810 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31562]: shutting down. 2024-09-15T04:01:56.810 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:56.811 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:01:56.811 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:01:56.813 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:01:58.883 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31630]: starting. 2024-09-15T04:01:58.884 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31630]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31629 2024-09-15T04:01:58.884 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31630]: created object 0... 2024-09-15T04:01:58.884 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31630]: created object 25... 2024-09-15T04:01:58.884 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31630]: created object 49... 2024-09-15T04:01:58.884 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31630]: finishing. 2024-09-15T04:01:58.884 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31630]: shutting down. 2024-09-15T04:01:59.812 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31631]: starting. 2024-09-15T04:01:59.812 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31631]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31629 2024-09-15T04:01:59.813 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31631]: shutting down. 2024-09-15T04:02:01.870 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:01.870 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31664]: starting. 2024-09-15T04:02:01.870 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31664]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31629 2024-09-15T04:02:01.870 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31664]: created object 0... 2024-09-15T04:02:01.870 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31664]: created object 25... 2024-09-15T04:02:01.870 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31664]: created object 49... 2024-09-15T04:02:01.870 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31664]: finishing. 2024-09-15T04:02:01.870 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31664]: shutting down. 2024-09-15T04:02:01.914 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:01.914 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31666]: starting. 2024-09-15T04:02:01.914 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31666]: listing objects. 2024-09-15T04:02:01.914 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31666]: listed object 0... 2024-09-15T04:02:01.915 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31666]: listed object 25... 2024-09-15T04:02:01.915 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31666]: saw 50 objects 2024-09-15T04:02:01.915 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31666]: shutting down. 2024-09-15T04:02:02.787 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:02.787 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31665]: starting. 2024-09-15T04:02:02.787 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31665]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31629 2024-09-15T04:02:02.787 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31665]: shutting down. 2024-09-15T04:02:02.788 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:02.788 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:02.788 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:02:02.791 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:02:05.018 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31733]: starting. 2024-09-15T04:02:05.018 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31733]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31732 2024-09-15T04:02:05.018 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31733]: created object 0... 2024-09-15T04:02:05.018 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31733]: created object 25... 2024-09-15T04:02:05.018 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31733]: created object 49... 2024-09-15T04:02:05.018 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31733]: finishing. 2024-09-15T04:02:05.018 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31733]: shutting down. 2024-09-15T04:02:05.843 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31734]: starting. 2024-09-15T04:02:05.844 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31734]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31732 2024-09-15T04:02:05.844 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31734]: shutting down. 2024-09-15T04:02:07.932 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:07.932 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31768]: starting. 2024-09-15T04:02:07.932 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31768]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31732 2024-09-15T04:02:07.932 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31768]: created object 0... 2024-09-15T04:02:07.932 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31768]: created object 25... 2024-09-15T04:02:07.933 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31768]: created object 49... 2024-09-15T04:02:07.933 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31768]: finishing. 2024-09-15T04:02:07.933 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31768]: shutting down. 2024-09-15T04:02:07.970 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:07.970 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31770]: starting. 2024-09-15T04:02:07.970 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31770]: listing objects. 2024-09-15T04:02:07.970 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31770]: listed object 0... 2024-09-15T04:02:07.970 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31770]: listed object 25... 2024-09-15T04:02:07.970 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31770]: saw 50 objects 2024-09-15T04:02:07.970 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31770]: shutting down. 2024-09-15T04:02:08.858 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:08.858 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31769]: starting. 2024-09-15T04:02:08.858 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31769]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31732 2024-09-15T04:02:08.858 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31769]: shutting down. 2024-09-15T04:02:08.859 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:08.859 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:08.859 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:02:08.862 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:02:10.971 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31837]: starting. 2024-09-15T04:02:10.971 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31837]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31836 2024-09-15T04:02:10.971 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31837]: created object 0... 2024-09-15T04:02:10.971 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31837]: created object 25... 2024-09-15T04:02:10.972 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31837]: created object 49... 2024-09-15T04:02:10.972 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31837]: finishing. 2024-09-15T04:02:10.972 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31837]: shutting down. 2024-09-15T04:02:11.784 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31838]: starting. 2024-09-15T04:02:11.784 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31838]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31836 2024-09-15T04:02:11.784 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31838]: shutting down. 2024-09-15T04:02:13.884 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:13.884 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31871]: starting. 2024-09-15T04:02:13.884 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31871]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31836 2024-09-15T04:02:13.884 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31871]: created object 0... 2024-09-15T04:02:13.884 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31871]: created object 25... 2024-09-15T04:02:13.884 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31871]: created object 49... 2024-09-15T04:02:13.884 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31871]: finishing. 2024-09-15T04:02:13.884 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31871]: shutting down. 2024-09-15T04:02:13.922 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:13.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31873]: starting. 2024-09-15T04:02:13.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31873]: listing objects. 2024-09-15T04:02:13.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31873]: listed object 0... 2024-09-15T04:02:13.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31873]: listed object 25... 2024-09-15T04:02:13.923 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31873]: saw 50 objects 2024-09-15T04:02:13.923 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31873]: shutting down. 2024-09-15T04:02:14.795 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:14.795 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31872]: starting. 2024-09-15T04:02:14.795 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31872]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31836 2024-09-15T04:02:14.795 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31872]: shutting down. 2024-09-15T04:02:14.796 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:14.796 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:14.796 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:02:14.798 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:02:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31940]: starting. 2024-09-15T04:02:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31940]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31939 2024-09-15T04:02:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31940]: created object 0... 2024-09-15T04:02:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31940]: created object 25... 2024-09-15T04:02:17.391 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31940]: created object 49... 2024-09-15T04:02:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31940]: finishing. 2024-09-15T04:02:17.392 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[31940]: shutting down. 2024-09-15T04:02:18.297 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31941]: starting. 2024-09-15T04:02:18.297 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31941]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31939 2024-09-15T04:02:18.297 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[31941]: shutting down. 2024-09-15T04:02:20.408 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:20.408 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31974]: starting. 2024-09-15T04:02:20.408 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31974]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-31939 2024-09-15T04:02:20.408 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31974]: created object 0... 2024-09-15T04:02:20.408 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31974]: created object 25... 2024-09-15T04:02:20.408 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31974]: created object 49... 2024-09-15T04:02:20.408 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31974]: finishing. 2024-09-15T04:02:20.408 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[31974]: shutting down. 2024-09-15T04:02:20.446 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:20.446 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31976]: starting. 2024-09-15T04:02:20.446 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31976]: listing objects. 2024-09-15T04:02:20.447 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31976]: listed object 0... 2024-09-15T04:02:20.447 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31976]: listed object 25... 2024-09-15T04:02:20.447 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31976]: saw 50 objects 2024-09-15T04:02:20.447 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[31976]: shutting down. 2024-09-15T04:02:21.313 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:21.313 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31975]: starting. 2024-09-15T04:02:21.313 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31975]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-31939 2024-09-15T04:02:21.313 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[31975]: shutting down. 2024-09-15T04:02:21.314 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:21.314 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:21.314 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:02:21.316 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:02:22.881 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32043]: starting. 2024-09-15T04:02:22.881 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32043]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32042 2024-09-15T04:02:22.881 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32043]: created object 0... 2024-09-15T04:02:22.881 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32043]: created object 25... 2024-09-15T04:02:22.881 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32043]: created object 49... 2024-09-15T04:02:22.881 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32043]: finishing. 2024-09-15T04:02:22.881 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32043]: shutting down. 2024-09-15T04:02:23.787 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32044]: starting. 2024-09-15T04:02:23.788 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32044]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32042 2024-09-15T04:02:23.788 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32044]: shutting down. 2024-09-15T04:02:25.867 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:25.867 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32077]: starting. 2024-09-15T04:02:25.867 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32077]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32042 2024-09-15T04:02:25.867 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32077]: created object 0... 2024-09-15T04:02:25.867 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32077]: created object 25... 2024-09-15T04:02:25.867 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32077]: created object 49... 2024-09-15T04:02:25.867 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32077]: finishing. 2024-09-15T04:02:25.867 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32077]: shutting down. 2024-09-15T04:02:25.906 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:25.906 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32079]: starting. 2024-09-15T04:02:25.906 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32079]: listing objects. 2024-09-15T04:02:25.906 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32079]: listed object 0... 2024-09-15T04:02:25.906 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32079]: listed object 25... 2024-09-15T04:02:25.906 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32079]: saw 50 objects 2024-09-15T04:02:25.906 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32079]: shutting down. 2024-09-15T04:02:26.795 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:26.795 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32078]: starting. 2024-09-15T04:02:26.795 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32078]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32042 2024-09-15T04:02:26.795 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32078]: shutting down. 2024-09-15T04:02:26.796 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:26.796 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:26.796 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:02:26.798 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:02:28.898 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32146]: starting. 2024-09-15T04:02:28.898 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32146]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32145 2024-09-15T04:02:28.898 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32146]: created object 0... 2024-09-15T04:02:28.898 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32146]: created object 25... 2024-09-15T04:02:28.899 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32146]: created object 49... 2024-09-15T04:02:28.899 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32146]: finishing. 2024-09-15T04:02:28.899 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32146]: shutting down. 2024-09-15T04:02:29.808 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32147]: starting. 2024-09-15T04:02:29.808 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32147]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32145 2024-09-15T04:02:29.808 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32147]: shutting down. 2024-09-15T04:02:31.873 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:31.873 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32180]: starting. 2024-09-15T04:02:31.873 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32180]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32145 2024-09-15T04:02:31.873 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32180]: created object 0... 2024-09-15T04:02:31.874 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32180]: created object 25... 2024-09-15T04:02:31.874 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32180]: created object 49... 2024-09-15T04:02:31.874 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32180]: finishing. 2024-09-15T04:02:31.874 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32180]: shutting down. 2024-09-15T04:02:31.911 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:31.911 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32182]: starting. 2024-09-15T04:02:31.911 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32182]: listing objects. 2024-09-15T04:02:31.912 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32182]: listed object 0... 2024-09-15T04:02:31.912 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32182]: listed object 25... 2024-09-15T04:02:31.912 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32182]: saw 50 objects 2024-09-15T04:02:31.912 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32182]: shutting down. 2024-09-15T04:02:32.796 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:32.796 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32181]: starting. 2024-09-15T04:02:32.796 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32181]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32145 2024-09-15T04:02:32.796 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32181]: shutting down. 2024-09-15T04:02:32.797 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:32.797 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:32.797 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:02:32.799 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:02:34.894 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32249]: starting. 2024-09-15T04:02:34.894 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32249]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32248 2024-09-15T04:02:34.894 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32249]: created object 0... 2024-09-15T04:02:34.895 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32249]: created object 25... 2024-09-15T04:02:34.895 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32249]: created object 49... 2024-09-15T04:02:34.895 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32249]: finishing. 2024-09-15T04:02:34.895 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32249]: shutting down. 2024-09-15T04:02:35.805 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32250]: starting. 2024-09-15T04:02:35.805 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32250]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32248 2024-09-15T04:02:35.805 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32250]: shutting down. 2024-09-15T04:02:37.924 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:37.924 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32283]: starting. 2024-09-15T04:02:37.924 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32283]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32248 2024-09-15T04:02:37.924 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32283]: created object 0... 2024-09-15T04:02:37.925 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32283]: created object 25... 2024-09-15T04:02:37.925 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32283]: created object 49... 2024-09-15T04:02:37.925 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32283]: finishing. 2024-09-15T04:02:37.925 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32283]: shutting down. 2024-09-15T04:02:37.964 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:37.964 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32285]: starting. 2024-09-15T04:02:37.964 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32285]: listing objects. 2024-09-15T04:02:37.964 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32285]: listed object 0... 2024-09-15T04:02:37.964 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32285]: listed object 25... 2024-09-15T04:02:37.964 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32285]: saw 50 objects 2024-09-15T04:02:37.964 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32285]: shutting down. 2024-09-15T04:02:38.823 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:38.823 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32284]: starting. 2024-09-15T04:02:38.823 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32284]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32248 2024-09-15T04:02:38.823 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32284]: shutting down. 2024-09-15T04:02:38.824 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:38.824 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:38.824 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:02:38.827 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:02:40.932 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32352]: starting. 2024-09-15T04:02:40.932 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32352]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32351 2024-09-15T04:02:40.933 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32352]: created object 0... 2024-09-15T04:02:40.933 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32352]: created object 25... 2024-09-15T04:02:40.933 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32352]: created object 49... 2024-09-15T04:02:40.933 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32352]: finishing. 2024-09-15T04:02:40.933 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32352]: shutting down. 2024-09-15T04:02:41.814 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32353]: starting. 2024-09-15T04:02:41.815 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32353]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32351 2024-09-15T04:02:41.815 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32353]: shutting down. 2024-09-15T04:02:43.941 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:43.941 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32386]: starting. 2024-09-15T04:02:43.941 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32386]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32351 2024-09-15T04:02:43.942 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32386]: created object 0... 2024-09-15T04:02:43.942 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32386]: created object 25... 2024-09-15T04:02:43.942 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32386]: created object 49... 2024-09-15T04:02:43.942 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32386]: finishing. 2024-09-15T04:02:43.942 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32386]: shutting down. 2024-09-15T04:02:43.980 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:43.980 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32388]: starting. 2024-09-15T04:02:43.980 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32388]: listing objects. 2024-09-15T04:02:43.980 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32388]: listed object 0... 2024-09-15T04:02:43.980 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32388]: listed object 25... 2024-09-15T04:02:43.980 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32388]: saw 50 objects 2024-09-15T04:02:43.980 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32388]: shutting down. 2024-09-15T04:02:44.755 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T04:02:44.756 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T04:02:44.756 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T04:02:44.756 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T04:02:44.759 INFO:tasks.ceph.mon.c:Started 2024-09-15T04:02:44.759 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T04:02:44.759 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:02:44.853 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:44.854 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32387]: starting. 2024-09-15T04:02:44.854 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32387]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32351 2024-09-15T04:02:44.854 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32387]: shutting down. 2024-09-15T04:02:44.854 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:44.855 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:44.855 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:02:44.857 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:02:44.867 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.865+0000 7fc26b6dd640 10 monclient: get_monmap_and_config 2024-09-15T04:02:44.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.865+0000 7fc26b6dd640 10 monclient: build_initial_monmap 2024-09-15T04:02:44.868 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.865+0000 7fc26b6dd640 10 monclient: monmap: 2024-09-15T04:02:44.868 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:44.868 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:44.868 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:02:44.867140+0000 2024-09-15T04:02:44.868 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:02:44.867140+0000 2024-09-15T04:02:44.868 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:44.868 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:44.868 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:02:44.868 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:02:44.868 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:02:44.868 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:02:44.869 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:02:44.869 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:44.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.865+0000 7fc26b6dd640 1 Processor -- start 2024-09-15T04:02:44.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.865+0000 7fc26b6dd640 1 -- start start 2024-09-15T04:02:44.869 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 10 monclient: init 2024-09-15T04:02:44.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:44.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-15T04:02:44.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fc2640a1bf0 0x7fc264131810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:44.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 10 monclient(hunting): picked mon.noname-c con 0x7fc2640a1bf0 addr v2:172.21.15.177:3300/0 2024-09-15T04:02:44.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc26409e780 addr v1:172.21.15.110:6789/0 2024-09-15T04:02:44.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc264136a40 0x7fc264138e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:44.871 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 10 monclient(hunting): picked mon.noname-a con 0x7fc264136a40 addr v2:172.21.15.110:3300/0 2024-09-15T04:02:44.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26409c9b0 con 0x7fc26409e780 2024-09-15T04:02:44.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:44.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2640a0c20 con 0x7fc264136a40 2024-09-15T04:02:44.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:44.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc26409cc90 con 0x7fc2640a1bf0 2024-09-15T04:02:44.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:44.872 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc26b6dd640 10 monclient(hunting): authenticate will time out at 988.511606s 2024-09-15T04:02:44.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc268c78640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc26409e780 0x7fc26409eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:38522/0 (socket says 172.21.15.110:38522) 2024-09-15T04:02:44.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc268c78640 1 -- 172.21.15.110:0/4276478217 learned_addr learned my addr 172.21.15.110:0/4276478217 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:02:44.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269c7a640 1 --2- 172.21.15.110:0/4276478217 >> v2:172.21.15.110:3300/0 conn(0x7fc264136a40 0x7fc264138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:44.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269479640 1 --2- 172.21.15.110:0/4276478217 >> v2:172.21.15.177:3300/0 conn(0x7fc2640a1bf0 0x7fc264131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:44.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269c7a640 10 monclient(hunting): get_auth_request con 0x7fc264136a40 auth_method 0 2024-09-15T04:02:44.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269c7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:44.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269c7a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:44.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269c7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:44.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269479640 10 monclient(hunting): get_auth_request con 0x7fc2640a1bf0 auth_method 0 2024-09-15T04:02:44.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269479640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:44.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269479640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:44.873 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269479640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:44.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269c7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:44.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269c7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:44.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269c7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:44.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269479640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:44.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269479640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:44.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.866+0000 7fc269479640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:44.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.867+0000 7fc269c7a640 10 monclient(hunting): handle_auth_done global_id 54630 payload 274 2024-09-15T04:02:44.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.867+0000 7fc269c7a640 1 -- 172.21.15.110:0/4276478217 >> v2:172.21.15.177:3300/0 conn(0x7fc2640a1bf0 msgr2=0x7fc264131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:44.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.867+0000 7fc269c7a640 1 --2- 172.21.15.110:0/4276478217 >> v2:172.21.15.177:3300/0 conn(0x7fc2640a1bf0 0x7fc264131810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:44.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.867+0000 7fc269c7a640 1 -- 172.21.15.110:0/4276478217 >> v1:172.21.15.110:6789/0 conn(0x7fc26409e780 legacy=0x7fc26409eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:44.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.867+0000 7fc269c7a640 10 monclient: _finish_hunting 0 2024-09-15T04:02:44.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.867+0000 7fc269c7a640 1 monclient: found mon.noname-a 2024-09-15T04:02:44.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.867+0000 7fc269c7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:44.874 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.867+0000 7fc269c7a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:02:44.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.867+0000 7fc269c7a640 1 -- 172.21.15.110:0/4276478217 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2640a1fc0 con 0x7fc264136a40 2024-09-15T04:02:44.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.869+0000 7fc269c7a640 1 -- 172.21.15.110:0/4276478217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc264136a40 msgr2=0x7fc264138e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:44.875 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.869+0000 7fc269c7a640 1 --2- 172.21.15.110:0/4276478217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc264136a40 0x7fc264138e20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc2640a2bb0 tx=0x7fc25c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e2aa4c5aad2866 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:44.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc25bfff640 1 -- 172.21.15.110:0/4276478217 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c007760 con 0x7fc264136a40 2024-09-15T04:02:44.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc25bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:44.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc25bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:02:44.881 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc25bfff640 10 monclient: dump: 2024-09-15T04:02:44.882 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:44.882 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:44.882 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:44.882 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:44.882 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:44.882 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:44.882 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:44.882 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:44.882 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:44.882 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:44.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc25bfff640 10 monclient: _finish_auth 0 2024-09-15T04:02:44.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc25bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:44.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc26b6dd640 5 monclient: authenticate success, global_id 54630 2024-09-15T04:02:44.882 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc26b6dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:02:44.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc25bfff640 1 -- 172.21.15.110:0/4276478217 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c02d050 con 0x7fc264136a40 2024-09-15T04:02:44.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc25bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:44.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc26b6dd640 10 monclient: get_monmap_and_config success 2024-09-15T04:02:44.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc26b6dd640 4 set_mon_vals no callback set 2024-09-15T04:02:44.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc25bfff640 1 -- 172.21.15.110:0/4276478217 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c007b30 con 0x7fc264136a40 2024-09-15T04:02:44.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc25bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:44.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc25bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:44.883 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc25bfff640 10 monclient: dump: 2024-09-15T04:02:44.883 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:44.883 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:44.883 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:44.883 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:44.883 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:44.884 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:44.884 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:44.884 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:44.884 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:44.884 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:44.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc26b6dd640 10 monclient: shutdown 2024-09-15T04:02:44.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc26b6dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:44.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc26b6dd640 1 -- 172.21.15.110:0/4276478217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc264136a40 msgr2=0x7fc264138e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:44.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc26b6dd640 1 --2- 172.21.15.110:0/4276478217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc264136a40 0x7fc264138e20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc2640a2bb0 tx=0x7fc25c009780 comp rx=0 tx=0).stop 2024-09-15T04:02:44.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc26b6dd640 1 -- 172.21.15.110:0/4276478217 shutdown_connections 2024-09-15T04:02:44.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc26b6dd640 1 --2- 172.21.15.110:0/4276478217 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc264136a40 0x7fc264138e20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:44.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc26b6dd640 1 --2- 172.21.15.110:0/4276478217 >> v2:172.21.15.177:3300/0 conn(0x7fc2640a1bf0 0x7fc264131810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:44.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc26b6dd640 1 -- 172.21.15.110:0/4276478217 >> 172.21.15.110:0/4276478217 conn(0x7fc264062dd0 msgr2=0x7fc2640631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:44.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc26b6dd640 1 -- 172.21.15.110:0/4276478217 shutdown_connections 2024-09-15T04:02:44.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.870+0000 7fc26b6dd640 1 -- 172.21.15.110:0/4276478217 wait complete. 2024-09-15T04:02:44.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.871+0000 7fc26b6dd640 10 monclient: build_initial_monmap 2024-09-15T04:02:44.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.871+0000 7fc26b6dd640 10 monclient: monmap: 2024-09-15T04:02:44.885 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:44.885 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:02:44.885 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:02:44.885 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:02:44.885 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:44.885 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:44.886 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:02:44.886 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:02:44.886 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:02:44.886 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:44.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.871+0000 7fc26b6dd640 1 Processor -- start 2024-09-15T04:02:44.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.871+0000 7fc26b6dd640 1 -- start start 2024-09-15T04:02:44.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.871+0000 7fc26b6dd640 10 monclient: init 2024-09-15T04:02:44.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:44.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T04:02:44.886 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc26409e780 0x7fc2641410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:44.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 10 monclient(hunting): picked mon.noname-a con 0x7fc26409e780 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:44.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2640a1bf0 0x7fc264141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:44.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 10 monclient(hunting): picked mon.noname-c con 0x7fc2640a1bf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:44.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc264136a40 0x7fc2640999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:44.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 10 monclient(hunting): picked mon.noname-b con 0x7fc264136a40 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:02:44.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:44.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26409c250 con 0x7fc26409e780 2024-09-15T04:02:44.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:44.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2640a2da0 con 0x7fc264136a40 2024-09-15T04:02:44.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:44.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26411c700 con 0x7fc2640a1bf0 2024-09-15T04:02:44.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:44.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc26b6dd640 10 monclient(hunting): authenticate will time out at 988.517290s 2024-09-15T04:02:44.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc26409e780 0x7fc2641410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:44.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc26409e780 0x7fc2641410e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34116/0 (socket says 172.21.15.110:34116) 2024-09-15T04:02:44.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc268c78640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2640a1bf0 0x7fc264141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:44.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 1 -- 172.21.15.110:0/3660745969 learned_addr learned my addr 172.21.15.110:0/3660745969 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:44.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 10 monclient(hunting): get_auth_request con 0x7fc26409e780 auth_method 0 2024-09-15T04:02:44.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:44.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:44.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:44.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc268c78640 10 monclient(hunting): get_auth_request con 0x7fc2640a1bf0 auth_method 0 2024-09-15T04:02:44.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc268c78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:44.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc268c78640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:44.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc268c78640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:44.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:44.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:44.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269c7a640 1 --2- 172.21.15.110:0/3660745969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc264136a40 0x7fc2640999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:44.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:44.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269c7a640 10 monclient(hunting): get_auth_request con 0x7fc264136a40 auth_method 0 2024-09-15T04:02:44.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269c7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:44.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269c7a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:44.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269c7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:44.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc268c78640 1 -- 172.21.15.110:0/3660745969 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2640a1bf0 msgr2=0x7fc264141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T04:02:44.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc268c78640 1 -- 172.21.15.110:0/3660745969 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2640a1bf0 msgr2=0x7fc264141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T04:02:44.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc268c78640 1 --2- 172.21.15.110:0/3660745969 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2640a1bf0 0x7fc264141620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:02:44.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc268c78640 1 --2- 172.21.15.110:0/3660745969 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2640a1bf0 0x7fc264141620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:02:44.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269c7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:44.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269c7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:44.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269c7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:44.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 10 monclient(hunting): handle_auth_done global_id 54633 payload 824 2024-09-15T04:02:44.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 1 -- 172.21.15.110:0/3660745969 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2640a1bf0 msgr2=0x7fc264141620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:02:44.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 1 --2- 172.21.15.110:0/3660745969 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2640a1bf0 0x7fc264141620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:44.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 1 -- 172.21.15.110:0/3660745969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc264136a40 msgr2=0x7fc2640999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:44.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 1 --2- 172.21.15.110:0/3660745969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc264136a40 0x7fc2640999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:44.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 10 monclient: _finish_hunting 0 2024-09-15T04:02:44.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 1 monclient: found mon.noname-a 2024-09-15T04:02:44.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:44.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:02:44.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.872+0000 7fc269479640 1 -- 172.21.15.110:0/3660745969 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c009a20 con 0x7fc26409e780 2024-09-15T04:02:44.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.874+0000 7fc269479640 1 -- 172.21.15.110:0/3660745969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc26409e780 msgr2=0x7fc2641410e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:44.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.874+0000 7fc269479640 1 --2- 172.21.15.110:0/3660745969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc26409e780 0x7fc2641410e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc26000ea20 tx=0x7fc2600090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:44.892 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc2597fa640 1 -- 172.21.15.110:0/3660745969 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26000cc20 con 0x7fc26409e780 2024-09-15T04:02:44.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc2597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:44.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc2597fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:02:44.893 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc2597fa640 10 monclient: dump: 2024-09-15T04:02:44.893 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:44.893 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:44.893 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:44.894 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:44.894 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:44.894 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:44.894 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:44.894 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:44.894 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:44.895 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:44.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc2597fa640 10 monclient: _finish_auth 0 2024-09-15T04:02:44.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc2597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:44.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc2597fa640 1 -- 172.21.15.110:0/3660745969 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26000cd80 con 0x7fc26409e780 2024-09-15T04:02:44.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc26b6dd640 5 monclient: authenticate success, global_id 54633 2024-09-15T04:02:44.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc2597fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:44.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc2597fa640 1 -- 172.21.15.110:0/3660745969 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260013680 con 0x7fc26409e780 2024-09-15T04:02:44.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc26b6dd640 10 monclient: _renew_subs 2024-09-15T04:02:44.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc26b6dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:44.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc26b6dd640 1 -- 172.21.15.110:0/3660745969 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26409b570 con 0x7fc26409e780 2024-09-15T04:02:44.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc2597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:44.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc2597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:44.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc2597fa640 10 monclient: dump: 2024-09-15T04:02:44.897 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:44.897 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:44.900 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:44.900 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:44.900 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:44.900 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:44.901 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:44.901 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:44.901 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:44.901 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:44.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc25b7fe640 4 set_mon_vals no callback set 2024-09-15T04:02:44.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc26b6dd640 10 monclient: _renew_subs 2024-09-15T04:02:44.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc26b6dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:44.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.875+0000 7fc26b6dd640 1 -- 172.21.15.110:0/3660745969 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc264141b60 con 0x7fc26409e780 2024-09-15T04:02:44.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.876+0000 7fc2597fa640 1 -- 172.21.15.110:0/3660745969 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fc260013a70 con 0x7fc26409e780 2024-09-15T04:02:44.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.876+0000 7fc26b6dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:44.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.876+0000 7fc26b6dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:44.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.876+0000 7fc26b6dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:44.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.876+0000 7fc26b6dd640 1 -- 172.21.15.110:0/3660745969 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26411cdd0 con 0x7fc26409e780 2024-09-15T04:02:44.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.876+0000 7fc2597fa640 1 --2- 172.21.15.110:0/3660745969 >> v2:172.21.15.177:6812/28201 conn(0x7fc248038730 0x7fc24803abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:44.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.876+0000 7fc2597fa640 1 -- 172.21.15.110:0/3660745969 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc26004b970 con 0x7fc26409e780 2024-09-15T04:02:44.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.876+0000 7fc268c78640 1 --2- 172.21.15.110:0/3660745969 >> v2:172.21.15.177:6812/28201 conn(0x7fc248038730 0x7fc24803abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:44.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.876+0000 7fc268c78640 10 monclient: get_auth_request con 0x7fc248038730 auth_method 0 2024-09-15T04:02:44.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.879+0000 7fc268c78640 1 --2- 172.21.15.110:0/3660745969 >> v2:172.21.15.177:6812/28201 conn(0x7fc248038730 0x7fc24803abe0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc2640a11a0 tx=0x7fc254009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:44.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.880+0000 7fc2597fa640 1 -- 172.21.15.110:0/3660745969 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fc2600123d0 con 0x7fc26409e780 2024-09-15T04:02:44.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.880+0000 7fc2597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:44.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:44.880+0000 7fc2597fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:02:45.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.099+0000 7fc26b6dd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:02:45.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.099+0000 7fc26b6dd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:02:45.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.099+0000 7fc26b6dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:45.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.099+0000 7fc26b6dd640 1 -- 172.21.15.110:0/3660745969 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc26409b8b0 con 0x7fc26409e780 2024-09-15T04:02:45.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.099+0000 7fc2597fa640 1 -- 172.21.15.110:0/3660745969 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fc260011e20 con 0x7fc26409e780 2024-09-15T04:02:45.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.099+0000 7fc2597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:02:45.101 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.099+0000 7fc2597fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:02:45.104 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":26,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":90,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:02:45.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.101+0000 7fc26b6dd640 1 -- 172.21.15.110:0/3660745969 >> v2:172.21.15.177:6812/28201 conn(0x7fc248038730 msgr2=0x7fc24803abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:45.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.101+0000 7fc26b6dd640 1 --2- 172.21.15.110:0/3660745969 >> v2:172.21.15.177:6812/28201 conn(0x7fc248038730 0x7fc24803abe0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc2640a11a0 tx=0x7fc254009040 comp rx=0 tx=0).stop 2024-09-15T04:02:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.101+0000 7fc26b6dd640 10 monclient: shutdown 2024-09-15T04:02:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.101+0000 7fc26b6dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.101+0000 7fc26b6dd640 1 -- 172.21.15.110:0/3660745969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc26409e780 msgr2=0x7fc2641410e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.101+0000 7fc26b6dd640 1 --2- 172.21.15.110:0/3660745969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc26409e780 0x7fc2641410e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc26000ea20 tx=0x7fc2600090f0 comp rx=0 tx=0).stop 2024-09-15T04:02:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.101+0000 7fc26b6dd640 1 -- 172.21.15.110:0/3660745969 shutdown_connections 2024-09-15T04:02:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.101+0000 7fc26b6dd640 1 --2- 172.21.15.110:0/3660745969 >> v2:172.21.15.177:6812/28201 conn(0x7fc248038730 0x7fc24803abe0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.101+0000 7fc26b6dd640 1 --2- 172.21.15.110:0/3660745969 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc264136a40 0x7fc2640999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.101+0000 7fc26b6dd640 1 --2- 172.21.15.110:0/3660745969 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fc2640a1bf0 0x7fc264141620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.101+0000 7fc26b6dd640 1 --2- 172.21.15.110:0/3660745969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc26409e780 0x7fc2641410e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.101+0000 7fc26b6dd640 1 -- 172.21.15.110:0/3660745969 >> 172.21.15.110:0/3660745969 conn(0x7fc264062dd0 msgr2=0x7fc2640a4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.101+0000 7fc26b6dd640 1 -- 172.21.15.110:0/3660745969 shutdown_connections 2024-09-15T04:02:45.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:45.101+0000 7fc26b6dd640 1 -- 172.21.15.110:0/3660745969 wait complete. 2024-09-15T04:02:46.932 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32455]: starting. 2024-09-15T04:02:46.932 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32455]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32454 2024-09-15T04:02:46.932 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32455]: created object 0... 2024-09-15T04:02:46.932 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32455]: created object 25... 2024-09-15T04:02:46.932 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32455]: created object 49... 2024-09-15T04:02:46.932 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32455]: finishing. 2024-09-15T04:02:46.932 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32455]: shutting down. 2024-09-15T04:02:47.860 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32456]: starting. 2024-09-15T04:02:47.861 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32456]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32454 2024-09-15T04:02:47.861 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32456]: shutting down. 2024-09-15T04:02:48.113 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:02:48.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.213+0000 7fcd7586f640 10 monclient: get_monmap_and_config 2024-09-15T04:02:48.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.213+0000 7fcd7586f640 10 monclient: build_initial_monmap 2024-09-15T04:02:48.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.213+0000 7fcd7586f640 10 monclient: monmap: 2024-09-15T04:02:48.214 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:48.214 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:48.214 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:02:48.214137+0000 2024-09-15T04:02:48.214 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:02:48.214137+0000 2024-09-15T04:02:48.214 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:48.214 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:48.214 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:02:48.214 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:02:48.214 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:02:48.214 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:02:48.215 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:02:48.215 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.213+0000 7fcd7586f640 1 Processor -- start 2024-09-15T04:02:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.213+0000 7fcd7586f640 1 -- start start 2024-09-15T04:02:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.213+0000 7fcd7586f640 10 monclient: init 2024-09-15T04:02:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd7586f640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd7586f640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-15T04:02:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd7586f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcd700a1a60 addr v1:172.21.15.177:6789/0 2024-09-15T04:02:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd7586f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcd70131640 0x7fcd70131a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:48.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd7586f640 10 monclient(hunting): picked mon.noname-a con 0x7fcd70131640 addr v2:172.21.15.110:3300/0 2024-09-15T04:02:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd7586f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fcd70131f80 0x7fcd70138b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd7586f640 10 monclient(hunting): picked mon.noname-c con 0x7fcd70131f80 addr v2:172.21.15.177:3300/0 2024-09-15T04:02:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd7586f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd700a1260 con 0x7fcd700a1a60 2024-09-15T04:02:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd7586f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd7586f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd7009c870 con 0x7fcd70131640 2024-09-15T04:02:48.218 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd7586f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd7586f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd700a0c60 con 0x7fcd70131f80 2024-09-15T04:02:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd7586f640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd7586f640 10 monclient(hunting): authenticate will time out at 991.858555s 2024-09-15T04:02:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd67fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcd70131640 0x7fcd70131a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd67fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcd70131640 0x7fcd70131a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34174/0 (socket says 172.21.15.110:34174) 2024-09-15T04:02:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd67fff640 1 -- 172.21.15.110:0/1015502369 learned_addr learned my addr 172.21.15.110:0/1015502369 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd67fff640 10 monclient(hunting): get_auth_request con 0x7fcd70131640 auth_method 0 2024-09-15T04:02:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd67fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:48.219 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd6f7fe640 1 --2- 172.21.15.110:0/1015502369 >> v2:172.21.15.177:3300/0 conn(0x7fcd70131f80 0x7fcd70138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd6f7fe640 10 monclient(hunting): get_auth_request con 0x7fcd70131f80 auth_method 0 2024-09-15T04:02:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd67fff640 10 monclient(hunting): handle_auth_done global_id 54660 payload 274 2024-09-15T04:02:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd67fff640 1 -- 172.21.15.110:0/1015502369 >> v2:172.21.15.177:3300/0 conn(0x7fcd70131f80 msgr2=0x7fcd70138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd67fff640 1 --2- 172.21.15.110:0/1015502369 >> v2:172.21.15.177:3300/0 conn(0x7fcd70131f80 0x7fcd70138b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.214+0000 7fcd67fff640 1 -- 172.21.15.110:0/1015502369 >> v1:172.21.15.177:6789/0 conn(0x7fcd700a1a60 legacy=0x7fcd7009d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.215+0000 7fcd67fff640 10 monclient: _finish_hunting 0 2024-09-15T04:02:48.220 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.215+0000 7fcd67fff640 1 monclient: found mon.noname-a 2024-09-15T04:02:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.215+0000 7fcd67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.215+0000 7fcd67fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:02:48.221 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.215+0000 7fcd67fff640 1 -- 172.21.15.110:0/1015502369 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7009cb10 con 0x7fcd70131640 2024-09-15T04:02:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.217+0000 7fcd67fff640 1 -- 172.21.15.110:0/1015502369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd70131640 msgr2=0x7fcd70131a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.217+0000 7fcd67fff640 1 --2- 172.21.15.110:0/1015502369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd70131640 0x7fcd70131a40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcd700a0bb0 tx=0x7fcd580091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=262de93f07a153ac server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd6e7fc640 1 -- 172.21.15.110:0/1015502369 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd58032440 con 0x7fcd70131640 2024-09-15T04:02:48.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd6e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:02:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd6e7fc640 10 monclient: dump: 2024-09-15T04:02:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:48.228 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd6e7fc640 10 monclient: _finish_auth 0 2024-09-15T04:02:48.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd6e7fc640 1 -- 172.21.15.110:0/1015502369 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd580325a0 con 0x7fcd70131640 2024-09-15T04:02:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd7586f640 5 monclient: authenticate success, global_id 54660 2024-09-15T04:02:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd7586f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:02:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd7586f640 10 monclient: get_monmap_and_config success 2024-09-15T04:02:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd6e7fc640 1 -- 172.21.15.110:0/1015502369 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd58032940 con 0x7fcd70131640 2024-09-15T04:02:48.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:48.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd7586f640 4 set_mon_vals no callback set 2024-09-15T04:02:48.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd6e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:48.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd6e7fc640 10 monclient: dump: 2024-09-15T04:02:48.230 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:48.230 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:48.230 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:48.230 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:48.230 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:48.230 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:48.230 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:48.230 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:48.230 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:48.230 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:48.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd7586f640 10 monclient: shutdown 2024-09-15T04:02:48.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd7586f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:48.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd7586f640 1 -- 172.21.15.110:0/1015502369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd70131640 msgr2=0x7fcd70131a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:48.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd7586f640 1 --2- 172.21.15.110:0/1015502369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd70131640 0x7fcd70131a40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcd700a0bb0 tx=0x7fcd580091d0 comp rx=0 tx=0).stop 2024-09-15T04:02:48.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd7586f640 1 -- 172.21.15.110:0/1015502369 shutdown_connections 2024-09-15T04:02:48.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd7586f640 1 --2- 172.21.15.110:0/1015502369 >> v2:172.21.15.177:3300/0 conn(0x7fcd70131f80 0x7fcd70138b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:48.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd7586f640 1 --2- 172.21.15.110:0/1015502369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd70131640 0x7fcd70131a40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:48.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd7586f640 1 -- 172.21.15.110:0/1015502369 >> 172.21.15.110:0/1015502369 conn(0x7fcd70063270 msgr2=0x7fcd70063670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:48.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd7586f640 1 -- 172.21.15.110:0/1015502369 shutdown_connections 2024-09-15T04:02:48.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.218+0000 7fcd7586f640 1 -- 172.21.15.110:0/1015502369 wait complete. 2024-09-15T04:02:48.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.219+0000 7fcd7586f640 10 monclient: build_initial_monmap 2024-09-15T04:02:48.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.219+0000 7fcd7586f640 10 monclient: monmap: 2024-09-15T04:02:48.232 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:48.232 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:02:48.232 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:02:48.232 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:02:48.232 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:48.232 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:48.233 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:02:48.233 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:02:48.233 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:02:48.233 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.219+0000 7fcd7586f640 1 Processor -- start 2024-09-15T04:02:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.219+0000 7fcd7586f640 1 -- start start 2024-09-15T04:02:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.219+0000 7fcd7586f640 10 monclient: init 2024-09-15T04:02:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T04:02:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd700a1a60 0x7fcd70137f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 10 monclient(hunting): picked mon.noname-a con 0x7fcd700a1a60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd70131640 0x7fcd70133c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 10 monclient(hunting): picked mon.noname-b con 0x7fcd70131640 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:02:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd70131f80 0x7fcd70134170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 10 monclient(hunting): picked mon.noname-c con 0x7fcd70131f80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:48.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd70137580 con 0x7fcd700a1a60 2024-09-15T04:02:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd701384a0 con 0x7fcd70131640 2024-09-15T04:02:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd700a1450 con 0x7fcd70131f80 2024-09-15T04:02:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd7586f640 10 monclient(hunting): authenticate will time out at 991.864082s 2024-09-15T04:02:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd700a1a60 0x7fcd70137f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd700a1a60 0x7fcd70137f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:34180/0 (socket says 172.21.15.110:34180) 2024-09-15T04:02:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 1 -- 172.21.15.110:0/2543549619 learned_addr learned my addr 172.21.15.110:0/2543549619 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 10 monclient(hunting): get_auth_request con 0x7fcd700a1a60 auth_method 0 2024-09-15T04:02:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:48.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6f7fe640 1 --2- 172.21.15.110:0/2543549619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd70131f80 0x7fcd70134170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6f7fe640 10 monclient(hunting): get_auth_request con 0x7fcd70131f80 auth_method 0 2024-09-15T04:02:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6f7fe640 1 -- 172.21.15.110:0/2543549619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd70131f80 msgr2=0x7fcd70134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T04:02:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6f7fe640 1 -- 172.21.15.110:0/2543549619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd70131f80 msgr2=0x7fcd70134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T04:02:48.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6f7fe640 1 --2- 172.21.15.110:0/2543549619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd70131f80 0x7fcd70134170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:02:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd67fff640 1 --2- 172.21.15.110:0/2543549619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd70131640 0x7fcd70133c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6f7fe640 1 --2- 172.21.15.110:0/2543549619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd70131f80 0x7fcd70134170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:02:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd67fff640 10 monclient(hunting): get_auth_request con 0x7fcd70131640 auth_method 0 2024-09-15T04:02:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd67fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 10 monclient(hunting): handle_auth_done global_id 54663 payload 824 2024-09-15T04:02:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 1 -- 172.21.15.110:0/2543549619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd70131f80 msgr2=0x7fcd70134170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:02:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 1 --2- 172.21.15.110:0/2543549619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd70131f80 0x7fcd70134170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 1 -- 172.21.15.110:0/2543549619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd70131640 msgr2=0x7fcd70133c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 1 --2- 172.21.15.110:0/2543549619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd70131640 0x7fcd70133c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 10 monclient: _finish_hunting 0 2024-09-15T04:02:48.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 1 monclient: found mon.noname-a 2024-09-15T04:02:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:02:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd6effd640 1 -- 172.21.15.110:0/2543549619 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd58009450 con 0x7fcd700a1a60 2024-09-15T04:02:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.220+0000 7fcd67fff640 1 --2- 172.21.15.110:0/2543549619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd70131640 0x7fcd70133c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T04:02:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd6effd640 1 -- 172.21.15.110:0/2543549619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd700a1a60 msgr2=0x7fcd70137f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd6effd640 1 --2- 172.21.15.110:0/2543549619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd700a1a60 0x7fcd70137f60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000c5e0 tx=0x7fcd60007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd66ffd640 1 -- 172.21.15.110:0/2543549619 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd60012840 con 0x7fcd700a1a60 2024-09-15T04:02:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd66ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:02:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd66ffd640 10 monclient: dump: 2024-09-15T04:02:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:48.238 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:48.239 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:48.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd66ffd640 10 monclient: _finish_auth 0 2024-09-15T04:02:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd66ffd640 1 -- 172.21.15.110:0/2543549619 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd600129a0 con 0x7fcd700a1a60 2024-09-15T04:02:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd7586f640 5 monclient: authenticate success, global_id 54663 2024-09-15T04:02:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd7586f640 10 monclient: _renew_subs 2024-09-15T04:02:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd7586f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd7586f640 1 -- 172.21.15.110:0/2543549619 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd700a3310 con 0x7fcd700a1a60 2024-09-15T04:02:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd66ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd66ffd640 1 -- 172.21.15.110:0/2543549619 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd60012de0 con 0x7fcd700a1a60 2024-09-15T04:02:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd66ffd640 10 monclient: dump: 2024-09-15T04:02:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:48.240 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:48.241 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd6dffb640 4 set_mon_vals no callback set 2024-09-15T04:02:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd7586f640 10 monclient: _renew_subs 2024-09-15T04:02:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd7586f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.222+0000 7fcd7586f640 1 -- 172.21.15.110:0/2543549619 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd70136340 con 0x7fcd700a1a60 2024-09-15T04:02:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.223+0000 7fcd7586f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:48.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.223+0000 7fcd7586f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.223+0000 7fcd7586f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.223+0000 7fcd7586f640 1 -- 172.21.15.110:0/2543549619 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd701365c0 con 0x7fcd700a1a60 2024-09-15T04:02:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.224+0000 7fcd66ffd640 1 -- 172.21.15.110:0/2543549619 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fcd60011590 con 0x7fcd700a1a60 2024-09-15T04:02:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.224+0000 7fcd66ffd640 1 --2- 172.21.15.110:0/2543549619 >> v2:172.21.15.177:6812/28201 conn(0x7fcd50038700 0x7fcd5003abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.224+0000 7fcd66ffd640 1 -- 172.21.15.110:0/2543549619 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fcd6004b760 con 0x7fcd700a1a60 2024-09-15T04:02:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.224+0000 7fcd67fff640 1 --2- 172.21.15.110:0/2543549619 >> v2:172.21.15.177:6812/28201 conn(0x7fcd50038700 0x7fcd5003abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.224+0000 7fcd67fff640 10 monclient: get_auth_request con 0x7fcd50038700 auth_method 0 2024-09-15T04:02:48.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.227+0000 7fcd67fff640 1 --2- 172.21.15.110:0/2543549619 >> v2:172.21.15.177:6812/28201 conn(0x7fcd50038700 0x7fcd5003abb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcd700a0a10 tx=0x7fcd58032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.228+0000 7fcd66ffd640 1 -- 172.21.15.110:0/2543549619 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fcd701365c0 con 0x7fcd700a1a60 2024-09-15T04:02:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.228+0000 7fcd66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:48.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.228+0000 7fcd66ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:02:48.441 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.440+0000 7fcd7586f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:02:48.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.440+0000 7fcd7586f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:02:48.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.440+0000 7fcd7586f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:48.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.440+0000 7fcd7586f640 1 -- 172.21.15.110:0/2543549619 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcd701365c0 con 0x7fcd700a1a60 2024-09-15T04:02:48.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.441+0000 7fcd66ffd640 1 -- 172.21.15.110:0/2543549619 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1534 (secure 0 0 0) 0x7fcd701365c0 con 0x7fcd700a1a60 2024-09-15T04:02:48.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.441+0000 7fcd66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:02:48.442 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.441+0000 7fcd66ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:02:48.442 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":26,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:02:48.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.443+0000 7fcd7586f640 1 -- 172.21.15.110:0/2543549619 >> v2:172.21.15.177:6812/28201 conn(0x7fcd50038700 msgr2=0x7fcd5003abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:48.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.443+0000 7fcd7586f640 1 --2- 172.21.15.110:0/2543549619 >> v2:172.21.15.177:6812/28201 conn(0x7fcd50038700 0x7fcd5003abb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcd700a0a10 tx=0x7fcd58032000 comp rx=0 tx=0).stop 2024-09-15T04:02:48.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.443+0000 7fcd7586f640 10 monclient: shutdown 2024-09-15T04:02:48.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.443+0000 7fcd7586f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:48.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.443+0000 7fcd7586f640 1 -- 172.21.15.110:0/2543549619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd700a1a60 msgr2=0x7fcd70137f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:48.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.443+0000 7fcd7586f640 1 --2- 172.21.15.110:0/2543549619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd700a1a60 0x7fcd70137f60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000c5e0 tx=0x7fcd60007ed0 comp rx=0 tx=0).stop 2024-09-15T04:02:48.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.443+0000 7fcd7586f640 1 -- 172.21.15.110:0/2543549619 shutdown_connections 2024-09-15T04:02:48.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.443+0000 7fcd7586f640 1 --2- 172.21.15.110:0/2543549619 >> v2:172.21.15.177:6812/28201 conn(0x7fcd50038700 0x7fcd5003abb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:48.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.443+0000 7fcd7586f640 1 --2- 172.21.15.110:0/2543549619 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcd70131f80 0x7fcd70134170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:48.444 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.443+0000 7fcd7586f640 1 --2- 172.21.15.110:0/2543549619 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcd70131640 0x7fcd70133c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:48.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.443+0000 7fcd7586f640 1 --2- 172.21.15.110:0/2543549619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcd700a1a60 0x7fcd70137f60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:48.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.443+0000 7fcd7586f640 1 -- 172.21.15.110:0/2543549619 >> 172.21.15.110:0/2543549619 conn(0x7fcd70063270 msgr2=0x7fcd70138900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:48.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.443+0000 7fcd7586f640 1 -- 172.21.15.110:0/2543549619 shutdown_connections 2024-09-15T04:02:48.445 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:48.443+0000 7fcd7586f640 1 -- 172.21.15.110:0/2543549619 wait complete. 2024-09-15T04:02:50.196 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:50.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32489]: starting. 2024-09-15T04:02:50.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32489]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32454 2024-09-15T04:02:50.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32489]: created object 0... 2024-09-15T04:02:50.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32489]: created object 25... 2024-09-15T04:02:50.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32489]: created object 49... 2024-09-15T04:02:50.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32489]: finishing. 2024-09-15T04:02:50.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32489]: shutting down. 2024-09-15T04:02:50.234 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:50.235 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32491]: starting. 2024-09-15T04:02:50.235 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32491]: listing objects. 2024-09-15T04:02:50.235 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32491]: listed object 0... 2024-09-15T04:02:50.235 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32491]: listed object 25... 2024-09-15T04:02:50.235 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32491]: saw 50 objects 2024-09-15T04:02:50.235 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32491]: shutting down. 2024-09-15T04:02:50.877 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:50.877 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32490]: starting. 2024-09-15T04:02:50.877 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32490]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32454 2024-09-15T04:02:50.878 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32490]: shutting down. 2024-09-15T04:02:50.878 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:50.878 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:50.878 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:02:50.881 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:02:51.455 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:02:51.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.558+0000 7fa076b9d640 10 monclient: get_monmap_and_config 2024-09-15T04:02:51.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.558+0000 7fa076b9d640 10 monclient: build_initial_monmap 2024-09-15T04:02:51.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.558+0000 7fa076b9d640 10 monclient: monmap: 2024-09-15T04:02:51.559 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:51.559 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:51.559 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:02:51.559167+0000 2024-09-15T04:02:51.559 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:02:51.559167+0000 2024-09-15T04:02:51.559 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:51.560 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.560 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:02:51.560 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:02:51.560 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:02:51.560 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:02:51.560 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:02:51.560 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.558+0000 7fa076b9d640 1 Processor -- start 2024-09-15T04:02:51.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa076b9d640 1 -- start start 2024-09-15T04:02:51.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa076b9d640 10 monclient: init 2024-09-15T04:02:51.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa076b9d640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:51.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa076b9d640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-15T04:02:51.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa076b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa0700a1a60 addr v1:172.21.15.110:6789/0 2024-09-15T04:02:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa076b9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fa070131640 0x7fa070131a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa076b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fa070131640 addr v2:172.21.15.177:3300/0 2024-09-15T04:02:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa076b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa070131f80 addr v1:172.21.15.177:6789/0 2024-09-15T04:02:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa076b9d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0700a1260 con 0x7fa0700a1a60 2024-09-15T04:02:51.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa076b9d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07009c870 con 0x7fa070131f80 2024-09-15T04:02:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa076b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa076b9d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0700a0c60 con 0x7fa070131640 2024-09-15T04:02:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa076b9d640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa076b9d640 10 monclient(hunting): authenticate will time out at 995.203663s 2024-09-15T04:02:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa074939640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa0700a1a60 0x7fa07009d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53760/0 (socket says 172.21.15.110:53760) 2024-09-15T04:02:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa074939640 1 -- 172.21.15.110:0/2404261615 learned_addr learned my addr 172.21.15.110:0/2404261615 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:02:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa06ffff640 1 --2- 172.21.15.110:0/2404261615 >> v2:172.21.15.177:3300/0 conn(0x7fa070131640 0x7fa070131a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa06ffff640 10 monclient(hunting): get_auth_request con 0x7fa070131640 auth_method 0 2024-09-15T04:02:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa06ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa06ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.559+0000 7fa06ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.560+0000 7fa06ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:51.564 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.560+0000 7fa06ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:51.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.560+0000 7fa06ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.561+0000 7fa074939640 1 -- 172.21.15.110:0/2404261615 >> v1:172.21.15.110:6789/0 conn(0x7fa0700a1a60 legacy=0x7fa07009d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.562+0000 7fa06f7fe640 1 -- 172.21.15.110:0/2404261615 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424245131 0 0) 0x7fa0700a1260 con 0x7fa0700a1a60 2024-09-15T04:02:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.562+0000 7fa06f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.562+0000 7fa06f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.562+0000 7fa06f7fe640 10 monclient(hunting): my global_id is 54675 2024-09-15T04:02:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.562+0000 7fa06f7fe640 1 -- 172.21.15.110:0/2404261615 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa05c0034e0 con 0x7fa0700a1a60 2024-09-15T04:02:51.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.562+0000 7fa07513a640 1 -- 172.21.15.110:0/2404261615 >> v1:172.21.15.177:6789/0 conn(0x7fa070131f80 legacy=0x7fa070138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.562+0000 7fa06ffff640 10 monclient(hunting): handle_auth_done global_id 5260 payload 274 2024-09-15T04:02:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.562+0000 7fa06ffff640 1 -- 172.21.15.110:0/2404261615 >> v1:172.21.15.177:6789/0 conn(0x7fa070131f80 legacy=0x7fa070138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.562+0000 7fa06ffff640 1 -- 172.21.15.110:0/2404261615 >> v1:172.21.15.110:6789/0 conn(0x7fa0700a1a60 legacy=0x7fa07009d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.562+0000 7fa06ffff640 10 monclient: _finish_hunting 0 2024-09-15T04:02:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.562+0000 7fa06ffff640 1 monclient: found mon.noname-c 2024-09-15T04:02:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.562+0000 7fa06ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.562+0000 7fa06ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T04:02:51.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.562+0000 7fa06ffff640 1 -- 172.21.15.110:0/2404261615 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07009cb10 con 0x7fa070131640 2024-09-15T04:02:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa06ffff640 1 -- 172.21.15.110:0/2404261615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa070131640 msgr2=0x7fa070131a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa06ffff640 1 --2- 172.21.15.110:0/2404261615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa070131640 0x7fa070131a40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa058009950 tx=0x7fa0580090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51f8a08aec06e3dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa06f7fe640 1 -- 172.21.15.110:0/2404261615 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05802aea0 con 0x7fa070131640 2024-09-15T04:02:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa06f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa06f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:02:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa06f7fe640 10 monclient: dump: 2024-09-15T04:02:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:51.569 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:51.570 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa06f7fe640 10 monclient: _finish_auth 0 2024-09-15T04:02:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa06f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa06f7fe640 1 -- 172.21.15.110:0/2404261615 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05802f4b0 con 0x7fa070131640 2024-09-15T04:02:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa076b9d640 5 monclient: authenticate success, global_id 5260 2024-09-15T04:02:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa076b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:02:51.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa06f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa06f7fe640 1 -- 172.21.15.110:0/2404261615 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05802f850 con 0x7fa070131640 2024-09-15T04:02:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa076b9d640 10 monclient: get_monmap_and_config success 2024-09-15T04:02:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa076b9d640 4 set_mon_vals no callback set 2024-09-15T04:02:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa06f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa06f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:02:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.565+0000 7fa06f7fe640 10 monclient: dump: 2024-09-15T04:02:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:51.571 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:51.572 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.566+0000 7fa076b9d640 10 monclient: shutdown 2024-09-15T04:02:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.566+0000 7fa076b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.566+0000 7fa076b9d640 1 -- 172.21.15.110:0/2404261615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa070131640 msgr2=0x7fa070131a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.566+0000 7fa076b9d640 1 --2- 172.21.15.110:0/2404261615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa070131640 0x7fa070131a40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa058009950 tx=0x7fa0580090f0 comp rx=0 tx=0).stop 2024-09-15T04:02:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.566+0000 7fa076b9d640 1 -- 172.21.15.110:0/2404261615 shutdown_connections 2024-09-15T04:02:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.566+0000 7fa076b9d640 1 --2- 172.21.15.110:0/2404261615 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa070131640 0x7fa070131a40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.566+0000 7fa076b9d640 1 -- 172.21.15.110:0/2404261615 >> 172.21.15.110:0/2404261615 conn(0x7fa070063270 msgr2=0x7fa070063670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.566+0000 7fa076b9d640 1 -- 172.21.15.110:0/2404261615 shutdown_connections 2024-09-15T04:02:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.566+0000 7fa076b9d640 1 -- 172.21.15.110:0/2404261615 wait complete. 2024-09-15T04:02:51.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.566+0000 7fa076b9d640 10 monclient: build_initial_monmap 2024-09-15T04:02:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.566+0000 7fa076b9d640 10 monclient: monmap: 2024-09-15T04:02:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:02:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:02:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:02:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:02:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:02:51.573 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:02:51.573 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 1 Processor -- start 2024-09-15T04:02:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 1 -- start start 2024-09-15T04:02:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 10 monclient: init 2024-09-15T04:02:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T04:02:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0700a1a60 0x7fa070140b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 10 monclient(hunting): picked mon.noname-b con 0x7fa0700a1a60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:02:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa070131640 0x7fa0701398d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 10 monclient(hunting): picked mon.noname-a con 0x7fa070131640 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa070131f80 0x7fa070134470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 10 monclient(hunting): picked mon.noname-c con 0x7fa070131f80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:51.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0701360d0 con 0x7fa070131640 2024-09-15T04:02:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0700a1450 con 0x7fa0700a1a60 2024-09-15T04:02:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0700a2f90 con 0x7fa070131f80 2024-09-15T04:02:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.567+0000 7fa076b9d640 10 monclient(hunting): authenticate will time out at 995.211963s 2024-09-15T04:02:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa070131640 0x7fa0701398d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa070131640 0x7fa0701398d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40328/0 (socket says 172.21.15.110:40328) 2024-09-15T04:02:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 1 -- 172.21.15.110:0/2000525734 learned_addr learned my addr 172.21.15.110:0/2000525734 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa07513a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa070131f80 0x7fa070134470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 10 monclient(hunting): get_auth_request con 0x7fa070131640 auth_method 0 2024-09-15T04:02:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:51.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa07513a640 10 monclient(hunting): get_auth_request con 0x7fa070131f80 auth_method 0 2024-09-15T04:02:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa07513a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa07513a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa07513a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa074939640 1 --2- 172.21.15.110:0/2000525734 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0700a1a60 0x7fa070140b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa07513a640 1 -- 172.21.15.110:0/2000525734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa070131f80 msgr2=0x7fa070134470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T04:02:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa07513a640 1 -- 172.21.15.110:0/2000525734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa070131f80 msgr2=0x7fa070134470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T04:02:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa07513a640 1 --2- 172.21.15.110:0/2000525734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa070131f80 0x7fa070134470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:02:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa074939640 10 monclient(hunting): get_auth_request con 0x7fa0700a1a60 auth_method 0 2024-09-15T04:02:51.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa074939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa074939640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa074939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa07513a640 1 --2- 172.21.15.110:0/2000525734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa070131f80 0x7fa070134470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:02:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa074939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa074939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa074939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 10 monclient(hunting): handle_auth_done global_id 54678 payload 824 2024-09-15T04:02:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 1 -- 172.21.15.110:0/2000525734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa070131f80 msgr2=0x7fa070134470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:02:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 1 --2- 172.21.15.110:0/2000525734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa070131f80 0x7fa070134470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 1 -- 172.21.15.110:0/2000525734 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0700a1a60 msgr2=0x7fa070140b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 1 --2- 172.21.15.110:0/2000525734 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0700a1a60 0x7fa070140b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 10 monclient: _finish_hunting 0 2024-09-15T04:02:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 1 monclient: found mon.noname-a 2024-09-15T04:02:51.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:02:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.568+0000 7fa06ffff640 1 -- 172.21.15.110:0/2000525734 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa058009390 con 0x7fa070131640 2024-09-15T04:02:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06ffff640 1 -- 172.21.15.110:0/2000525734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa070131640 msgr2=0x7fa0701398d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06ffff640 1 --2- 172.21.15.110:0/2000525734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa070131640 0x7fa0701398d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa0580099b0 tx=0x7fa058032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06cff9640 1 -- 172.21.15.110:0/2000525734 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05802aea0 con 0x7fa070131640 2024-09-15T04:02:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:02:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06cff9640 10 monclient: dump: 2024-09-15T04:02:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.583 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:51.584 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06cff9640 10 monclient: _finish_auth 0 2024-09-15T04:02:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa076b9d640 5 monclient: authenticate success, global_id 54678 2024-09-15T04:02:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06cff9640 1 -- 172.21.15.110:0/2000525734 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0580328e0 con 0x7fa070131640 2024-09-15T04:02:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06cff9640 1 -- 172.21.15.110:0/2000525734 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa058032ca0 con 0x7fa070131640 2024-09-15T04:02:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa076b9d640 10 monclient: _renew_subs 2024-09-15T04:02:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa076b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:51.584 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa076b9d640 1 -- 172.21.15.110:0/2000525734 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa070137b00 con 0x7fa070131640 2024-09-15T04:02:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06effd640 4 set_mon_vals no callback set 2024-09-15T04:02:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa06cff9640 10 monclient: dump: 2024-09-15T04:02:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:51.585 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:51.585 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa076b9d640 10 monclient: _renew_subs 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa076b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.570+0000 7fa076b9d640 1 -- 172.21.15.110:0/2000525734 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa070136c10 con 0x7fa070131640 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.571+0000 7fa06cff9640 1 -- 172.21.15.110:0/2000525734 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fa05803c650 con 0x7fa070131640 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.571+0000 7fa076b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.571+0000 7fa076b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.571+0000 7fa076b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.571+0000 7fa076b9d640 1 -- 172.21.15.110:0/2000525734 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa070137f00 con 0x7fa070131640 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.572+0000 7fa06cff9640 1 --2- 172.21.15.110:0/2000525734 >> v2:172.21.15.177:6812/28201 conn(0x7fa050038700 0x7fa05003abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.572+0000 7fa06cff9640 1 -- 172.21.15.110:0/2000525734 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa05806be20 con 0x7fa070131640 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.572+0000 7fa074939640 1 --2- 172.21.15.110:0/2000525734 >> v2:172.21.15.177:6812/28201 conn(0x7fa050038700 0x7fa05003abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.572+0000 7fa074939640 10 monclient: get_auth_request con 0x7fa050038700 auth_method 0 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.574+0000 7fa074939640 1 --2- 172.21.15.110:0/2000525734 >> v2:172.21.15.177:6812/28201 conn(0x7fa050038700 0x7fa05003abb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa060005bc0 tx=0x7fa060009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.576+0000 7fa06cff9640 1 -- 172.21.15.110:0/2000525734 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fa058032e40 con 0x7fa070131640 2024-09-15T04:02:51.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.576+0000 7fa06cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:51.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.576+0000 7fa06cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:02:51.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.794+0000 7fa076b9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:02:51.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.794+0000 7fa076b9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:02:51.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.794+0000 7fa076b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:51.795 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.794+0000 7fa076b9d640 1 -- 172.21.15.110:0/2000525734 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa070070950 con 0x7fa070131640 2024-09-15T04:02:51.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.794+0000 7fa06cff9640 1 -- 172.21.15.110:0/2000525734 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7fa058037020 con 0x7fa070131640 2024-09-15T04:02:51.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.794+0000 7fa06cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:02:51.796 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.794+0000 7fa06cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:02:51.796 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:02:51.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.797+0000 7fa076b9d640 1 -- 172.21.15.110:0/2000525734 >> v2:172.21.15.177:6812/28201 conn(0x7fa050038700 msgr2=0x7fa05003abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:51.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.797+0000 7fa076b9d640 1 --2- 172.21.15.110:0/2000525734 >> v2:172.21.15.177:6812/28201 conn(0x7fa050038700 0x7fa05003abb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa060005bc0 tx=0x7fa060009040 comp rx=0 tx=0).stop 2024-09-15T04:02:51.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.797+0000 7fa076b9d640 10 monclient: shutdown 2024-09-15T04:02:51.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.797+0000 7fa076b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:51.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.797+0000 7fa076b9d640 1 -- 172.21.15.110:0/2000525734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa070131640 msgr2=0x7fa0701398d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:51.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.797+0000 7fa076b9d640 1 --2- 172.21.15.110:0/2000525734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa070131640 0x7fa0701398d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa0580099b0 tx=0x7fa058032000 comp rx=0 tx=0).stop 2024-09-15T04:02:51.798 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.797+0000 7fa076b9d640 1 -- 172.21.15.110:0/2000525734 shutdown_connections 2024-09-15T04:02:51.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.797+0000 7fa076b9d640 1 --2- 172.21.15.110:0/2000525734 >> v2:172.21.15.177:6812/28201 conn(0x7fa050038700 0x7fa05003abb0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:51.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.797+0000 7fa076b9d640 1 --2- 172.21.15.110:0/2000525734 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fa070131f80 0x7fa070134470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:51.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.797+0000 7fa076b9d640 1 --2- 172.21.15.110:0/2000525734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa070131640 0x7fa0701398d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:51.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.797+0000 7fa076b9d640 1 --2- 172.21.15.110:0/2000525734 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa0700a1a60 0x7fa070140b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:51.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.797+0000 7fa076b9d640 1 -- 172.21.15.110:0/2000525734 >> 172.21.15.110:0/2000525734 conn(0x7fa070063270 msgr2=0x7fa0700a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:51.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.797+0000 7fa076b9d640 1 -- 172.21.15.110:0/2000525734 shutdown_connections 2024-09-15T04:02:51.799 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.797+0000 7fa076b9d640 1 -- 172.21.15.110:0/2000525734 wait complete. 2024-09-15T04:02:51.808 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T04:02:51.809 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T04:02:51.809 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T04:02:51.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.903+0000 7f4e0b480640 10 monclient: get_monmap_and_config 2024-09-15T04:02:51.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.903+0000 7f4e0b480640 10 monclient: build_initial_monmap 2024-09-15T04:02:51.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.903+0000 7f4e0b480640 10 monclient: monmap: 2024-09-15T04:02:51.904 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:51.904 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:51.904 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:02:51.904044+0000 2024-09-15T04:02:51.904 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:02:51.904044+0000 2024-09-15T04:02:51.904 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:51.904 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.904 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:02:51.904 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:02:51.905 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:02:51.905 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:02:51.905 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:02:51.905 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.903+0000 7f4e0b480640 1 Processor -- start 2024-09-15T04:02:51.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.903+0000 7f4e0b480640 1 -- start start 2024-09-15T04:02:51.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.903+0000 7f4e0b480640 10 monclient: init 2024-09-15T04:02:51.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0b480640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:51.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0b480640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T04:02:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0b480640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4e040a1a10 0x7f4e04131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0b480640 10 monclient(hunting): picked mon.noname-c con 0x7f4e040a1a10 addr v2:172.21.15.177:3300/0 2024-09-15T04:02:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0b480640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e0409e770 addr v1:172.21.15.110:6789/0 2024-09-15T04:02:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0b480640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e041369e0 addr v1:172.21.15.177:6789/0 2024-09-15T04:02:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0b480640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0409f2c0 con 0x7f4e0409e770 2024-09-15T04:02:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0b480640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e040a0850 con 0x7f4e041369e0 2024-09-15T04:02:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0b480640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0b480640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e0409c9a0 con 0x7f4e040a1a10 2024-09-15T04:02:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0b480640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0b480640 10 monclient(hunting): authenticate will time out at 995.548330s 2024-09-15T04:02:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e08a1b640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4e0409e770 0x7f4e0409ebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53770/0 (socket says 172.21.15.110:53770) 2024-09-15T04:02:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e08a1b640 1 -- 172.21.15.110:0/2915468941 learned_addr learned my addr 172.21.15.110:0/2915468941 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:02:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0921c640 1 --2- 172.21.15.110:0/2915468941 >> v2:172.21.15.177:3300/0 conn(0x7f4e040a1a10 0x7f4e04131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.910 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0921c640 10 monclient(hunting): get_auth_request con 0x7f4e040a1a10 auth_method 0 2024-09-15T04:02:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0921c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0921c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0921c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0921c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0921c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.904+0000 7f4e0921c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.906+0000 7f4e08a1b640 1 -- 172.21.15.110:0/2915468941 >> v1:172.21.15.110:6789/0 conn(0x7f4e0409e770 legacy=0x7f4e0409ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.906+0000 7f4e03fff640 1 -- 172.21.15.110:0/2915468941 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160490560 0 0) 0x7f4e0409f2c0 con 0x7f4e0409e770 2024-09-15T04:02:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.906+0000 7f4e03fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.906+0000 7f4e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:51.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.906+0000 7f4e03fff640 10 monclient(hunting): my global_id is 54681 2024-09-15T04:02:51.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.906+0000 7f4e03fff640 1 -- 172.21.15.110:0/2915468941 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dec0034e0 con 0x7f4e0409e770 2024-09-15T04:02:51.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.906+0000 7f4e09a1d640 1 -- 172.21.15.110:0/2915468941 >> v1:172.21.15.177:6789/0 conn(0x7f4e041369e0 legacy=0x7f4e04138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:51.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.906+0000 7f4e03fff640 1 -- 172.21.15.110:0/2915468941 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3104340505 0 0) 0x7f4dec0034e0 con 0x7f4e0409e770 2024-09-15T04:02:51.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.906+0000 7f4e03fff640 1 -- 172.21.15.110:0/2915468941 >> v2:172.21.15.177:3300/0 conn(0x7f4e040a1a10 msgr2=0x7f4e04131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:51.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.906+0000 7f4e03fff640 1 --2- 172.21.15.110:0/2915468941 >> v2:172.21.15.177:3300/0 conn(0x7f4e040a1a10 0x7f4e04131b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:51.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.906+0000 7f4e03fff640 1 -- 172.21.15.110:0/2915468941 >> v1:172.21.15.177:6789/0 conn(0x7f4e041369e0 legacy=0x7f4e04138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:51.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient: _finish_hunting 0 2024-09-15T04:02:51.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 1 monclient: found mon.noname-a-legacy 2024-09-15T04:02:51.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:51.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.110:6789/0 2024-09-15T04:02:51.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 1 -- 172.21.15.110:0/2915468941 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0409cc80 con 0x7f4e0409e770 2024-09-15T04:02:51.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient: _finish_auth 0 2024-09-15T04:02:51.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:51.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0b480640 5 monclient: authenticate success, global_id 54681 2024-09-15T04:02:51.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 1 -- 172.21.15.110:0/2915468941 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3165669920 0 0) 0x7f4df80017e0 con 0x7f4e0409e770 2024-09-15T04:02:51.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0b480640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:02:51.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:51.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T04:02:51.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient: dump: 2024-09-15T04:02:51.913 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:51.913 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:51.913 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.913 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.913 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:51.913 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.913 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:51.914 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:51.914 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:51.914 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] but i'm connected to v1:172.21.15.110:6789/0, reconnecting 2024-09-15T04:02:51.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:51.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 1 -- 172.21.15.110:0/2915468941 >> v1:172.21.15.110:6789/0 conn(0x7f4e0409e770 legacy=0x7f4e0409ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:51.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T04:02:51.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 1 --2- 172.21.15.110:0/2915468941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dec003a50 0x7f4dec003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient(hunting): picked mon.a con 0x7f4dec003a50 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:51.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 1 --2- 172.21.15.110:0/2915468941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dec004780 0x7f4dec004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient(hunting): picked mon.c con 0x7f4dec004780 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:51.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 1 --2- 172.21.15.110:0/2915468941 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4dec005230 0x7f4dec00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient(hunting): picked mon.b con 0x7f4dec005230 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:02:51.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:51.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 1 -- 172.21.15.110:0/2915468941 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0409c9a0 con 0x7f4dec003a50 2024-09-15T04:02:51.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:51.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 1 -- 172.21.15.110:0/2915468941 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec00c1f0 con 0x7f4dec005230 2024-09-15T04:02:51.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:51.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 1 -- 172.21.15.110:0/2915468941 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dec00c3b0 con 0x7f4dec004780 2024-09-15T04:02:51.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:51.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e03fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T04:02:51.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0b480640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T04:02:51.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 1 --2- 172.21.15.110:0/2915468941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dec003a50 0x7f4dec003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 10 monclient(hunting): get_auth_request con 0x7f4dec003a50 auth_method 0 2024-09-15T04:02:51.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:51.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:51.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e08a1b640 1 --2- 172.21.15.110:0/2915468941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dec004780 0x7f4dec004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:02:51.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e08a1b640 10 monclient(hunting): get_auth_request con 0x7f4dec004780 auth_method 0 2024-09-15T04:02:51.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e08a1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:51.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e08a1b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:51.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e08a1b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:02:51.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:51.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:51.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:02:51.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e09a1d640 1 --2- 172.21.15.110:0/2915468941 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4dec005230 0x7f4dec00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e09a1d640 10 monclient(hunting): get_auth_request con 0x7f4dec005230 auth_method 0 2024-09-15T04:02:51.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e09a1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:51.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e09a1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:51.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e09a1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:02:51.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e08a1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:51.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e08a1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:51.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e08a1b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:02:51.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 10 monclient(hunting): handle_auth_done global_id 54681 payload 293 2024-09-15T04:02:51.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 1 -- 172.21.15.110:0/2915468941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dec004780 msgr2=0x7f4dec004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:51.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 1 --2- 172.21.15.110:0/2915468941 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4dec004780 0x7f4dec004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:51.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 1 -- 172.21.15.110:0/2915468941 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4dec005230 msgr2=0x7f4dec00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:51.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 1 --2- 172.21.15.110:0/2915468941 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4dec005230 0x7f4dec00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:51.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 10 monclient: _finish_hunting 0 2024-09-15T04:02:51.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 1 monclient: found mon.a 2024-09-15T04:02:51.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:51.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:51.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 1 -- 172.21.15.110:0/2915468941 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dec0034e0 con 0x7f4dec003a50 2024-09-15T04:02:51.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 10 monclient: _finish_auth 0 2024-09-15T04:02:51.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e0921c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:51.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e09a1d640 1 --2- 172.21.15.110:0/2915468941 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4dec005230 0x7f4dec00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T04:02:51.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.907+0000 7f4e09a1d640 1 -- 172.21.15.110:0/2915468941 reap_dead start 2024-09-15T04:02:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e0921c640 1 -- 172.21.15.110:0/2915468941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dec003a50 msgr2=0x7f4dec003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e0921c640 1 --2- 172.21.15.110:0/2915468941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dec003a50 0x7f4dec003e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4df400e210 tx=0x7f4df400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bebdad43328070db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e03fff640 1 -- 172.21.15.110:0/2915468941 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df402c040 con 0x7f4dec003a50 2024-09-15T04:02:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e03fff640 10 monclient: dump: 2024-09-15T04:02:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.927 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:51.928 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e0b480640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:02:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e03fff640 1 -- 172.21.15.110:0/2915468941 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df400f040 con 0x7f4dec003a50 2024-09-15T04:02:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e03fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e03fff640 1 -- 172.21.15.110:0/2915468941 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4030840 con 0x7f4dec003a50 2024-09-15T04:02:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e0b480640 10 monclient: get_monmap_and_config success 2024-09-15T04:02:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e0b480640 4 set_mon_vals no callback set 2024-09-15T04:02:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:51.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e03fff640 10 monclient: dump: 2024-09-15T04:02:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:51.929 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e0b480640 10 monclient: shutdown 2024-09-15T04:02:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e0b480640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:51.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e0b480640 1 -- 172.21.15.110:0/2915468941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dec003a50 msgr2=0x7f4dec003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e0b480640 1 --2- 172.21.15.110:0/2915468941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dec003a50 0x7f4dec003e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4df400e210 tx=0x7f4df400b040 comp rx=0 tx=0).stop 2024-09-15T04:02:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e0b480640 1 -- 172.21.15.110:0/2915468941 shutdown_connections 2024-09-15T04:02:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e0b480640 1 --2- 172.21.15.110:0/2915468941 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4dec003a50 0x7f4dec003e60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.910+0000 7f4e0b480640 1 -- 172.21.15.110:0/2915468941 >> 172.21.15.110:0/2915468941 conn(0x7f4e0406e860 msgr2=0x7f4e0406ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.911+0000 7f4e0b480640 1 -- 172.21.15.110:0/2915468941 shutdown_connections 2024-09-15T04:02:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.911+0000 7f4e0b480640 1 -- 172.21.15.110:0/2915468941 wait complete. 2024-09-15T04:02:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.911+0000 7f4e0b480640 10 monclient: build_initial_monmap 2024-09-15T04:02:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.911+0000 7f4e0b480640 10 monclient: monmap: 2024-09-15T04:02:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:02:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:02:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:02:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:51.930 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:02:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:02:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:02:51.931 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.911+0000 7f4e0b480640 1 Processor -- start 2024-09-15T04:02:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.911+0000 7f4e0b480640 1 -- start start 2024-09-15T04:02:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.911+0000 7f4e0b480640 10 monclient: init 2024-09-15T04:02:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T04:02:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4dec003a50 0x7f4e041305b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 10 monclient(hunting): picked mon.noname-b con 0x7f4dec003a50 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:02:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e04130af0 0x7f4e04099700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 10 monclient(hunting): picked mon.noname-c con 0x7f4e04130af0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:51.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e04099c40 0x7f4e0409a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 10 monclient(hunting): picked mon.noname-a con 0x7f4e04099c40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e04130ee0 con 0x7f4e04099c40 2024-09-15T04:02:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e040a11c0 con 0x7f4dec003a50 2024-09-15T04:02:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e040a46f0 con 0x7f4e04130af0 2024-09-15T04:02:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0b480640 10 monclient(hunting): authenticate will time out at 995.556384s 2024-09-15T04:02:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e08a1b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e04130af0 0x7f4e04099700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e04099c40 0x7f4e0409a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e08a1b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e04130af0 0x7f4e04099700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52154/0 (socket says 172.21.15.110:52154) 2024-09-15T04:02:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e08a1b640 1 -- 172.21.15.110:0/482496775 learned_addr learned my addr 172.21.15.110:0/482496775 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:51.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e08a1b640 10 monclient(hunting): get_auth_request con 0x7f4e04130af0 auth_method 0 2024-09-15T04:02:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e08a1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e08a1b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e08a1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 10 monclient(hunting): get_auth_request con 0x7f4e04099c40 auth_method 0 2024-09-15T04:02:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e08a1b640 1 -- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e04130af0 msgr2=0x7f4e04099700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T04:02:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e08a1b640 1 -- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e04130af0 msgr2=0x7f4e04099700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T04:02:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e08a1b640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e04130af0 0x7f4e04099700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:02:51.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e08a1b640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e04130af0 0x7f4e04099700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:02:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0921c640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4dec003a50 0x7f4e041305b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 10 monclient(hunting): handle_auth_done global_id 54684 payload 824 2024-09-15T04:02:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 1 -- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e04130af0 msgr2=0x7f4e04099700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:02:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4e04130af0 0x7f4e04099700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 1 -- 172.21.15.110:0/482496775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4dec003a50 msgr2=0x7f4e041305b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4dec003a50 0x7f4e041305b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 10 monclient: _finish_hunting 0 2024-09-15T04:02:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 1 monclient: found mon.noname-a 2024-09-15T04:02:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:02:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e09a1d640 1 -- 172.21.15.110:0/482496775 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0411c750 con 0x7f4e04099c40 2024-09-15T04:02:51.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0921c640 10 monclient: get_auth_request con 0x7f4dec003a50 auth_method 0 2024-09-15T04:02:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.912+0000 7f4e0921c640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4dec003a50 0x7f4e041305b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:02:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.914+0000 7f4e09a1d640 1 -- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e04099c40 msgr2=0x7f4e0409a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e09a1d640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e04099c40 0x7f4e0409a0b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4e04132700 tx=0x7f4dfc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e017fa640 1 -- 172.21.15.110:0/482496775 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc0074f0 con 0x7f4e04099c40 2024-09-15T04:02:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e017fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:02:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e017fa640 10 monclient: dump: 2024-09-15T04:02:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.935 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:51.936 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e017fa640 10 monclient: _finish_auth 0 2024-09-15T04:02:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e0b480640 5 monclient: authenticate success, global_id 54684 2024-09-15T04:02:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e017fa640 1 -- 172.21.15.110:0/482496775 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dfc005780 con 0x7f4e04099c40 2024-09-15T04:02:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e017fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e017fa640 1 -- 172.21.15.110:0/482496775 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc005b40 con 0x7f4e04099c40 2024-09-15T04:02:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e0b480640 10 monclient: _renew_subs 2024-09-15T04:02:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e0b480640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e0b480640 1 -- 172.21.15.110:0/482496775 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0411cc80 con 0x7f4e04099c40 2024-09-15T04:02:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:51.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e017fa640 10 monclient: dump: 2024-09-15T04:02:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:51.937 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e037fe640 4 set_mon_vals no callback set 2024-09-15T04:02:51.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e0b480640 10 monclient: _renew_subs 2024-09-15T04:02:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e0b480640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.915+0000 7f4e0b480640 1 -- 172.21.15.110:0/482496775 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0409b9a0 con 0x7f4e04099c40 2024-09-15T04:02:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e0b480640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e0b480640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e0409a5f0 0x7f4e04095a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e0b480640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e0b480640 1 -- 172.21.15.110:0/482496775 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e04063e80 con 0x7f4e0409a5f0 2024-09-15T04:02:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e0921c640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e0409a5f0 0x7f4e04095a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e0921c640 10 monclient: get_auth_request con 0x7f4e0409a5f0 auth_method 0 2024-09-15T04:02:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e0921c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e0921c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e0921c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e017fa640 1 -- 172.21.15.110:0/482496775 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f4dfc00e5b0 con 0x7f4e04099c40 2024-09-15T04:02:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e0921c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:51.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e0921c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e0921c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e0921c640 10 monclient(hunting): handle_auth_done global_id 54687 payload 824 2024-09-15T04:02:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e017fa640 1 --2- 172.21.15.110:0/482496775 >> v2:172.21.15.177:6812/28201 conn(0x7f4df0038730 0x7f4df003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e0921c640 1 -- 172.21.15.110:0/482496775 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e0411c8e0 con 0x7f4e0409a5f0 2024-09-15T04:02:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e017fa640 1 -- 172.21.15.110:0/482496775 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f4dfc04a060 con 0x7f4e04099c40 2024-09-15T04:02:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e08a1b640 1 --2- 172.21.15.110:0/482496775 >> v2:172.21.15.177:6812/28201 conn(0x7f4df0038730 0x7f4df003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.916+0000 7f4e08a1b640 10 monclient: get_auth_request con 0x7f4df0038730 auth_method 0 2024-09-15T04:02:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.918+0000 7f4e0921c640 1 -- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e0409a5f0 msgr2=0x7f4e04095a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.918+0000 7f4e0921c640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e0409a5f0 0x7f4e04095a70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4df4035000 tx=0x7f4df4036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.918+0000 7f4e017fa640 1 -- 172.21.15.110:0/482496775 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df403f070 con 0x7f4e0409a5f0 2024-09-15T04:02:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.918+0000 7f4e017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.918+0000 7f4e017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.918+0000 7f4e017fa640 10 monclient: dump: 2024-09-15T04:02:51.939 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:51.940 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.919+0000 7f4e017fa640 1 -- 172.21.15.110:0/482496775 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f4df4039840 con 0x7f4e0409a5f0 2024-09-15T04:02:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.919+0000 7f4e017fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.919+0000 7f4e017fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:02:51.940 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.919+0000 7f4e017fa640 1 -- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e0409a5f0 msgr2=0x7f4e04095a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.919+0000 7f4e017fa640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e0409a5f0 0x7f4e04095a70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4df4035000 tx=0x7f4df4036000 comp rx=0 tx=0).stop 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.919+0000 7f4e017fa640 1 -- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e0409a5f0 msgr2=0x7f4e04095a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.919+0000 7f4e017fa640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e0409a5f0 0x7f4e04095a70 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4df4035000 tx=0x7f4df4036000 comp rx=0 tx=0).stop 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.919+0000 7f4e08a1b640 1 --2- 172.21.15.110:0/482496775 >> v2:172.21.15.177:6812/28201 conn(0x7f4df0038730 0x7f4df003abe0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4df8001990 tx=0x7f4df8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.924+0000 7f4e0b480640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.924+0000 7f4e0b480640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e04095fb0 0x7f4e04096380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.924+0000 7f4e0b480640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.924+0000 7f4e0b480640 1 -- 172.21.15.110:0/482496775 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e04137280 con 0x7f4e04095fb0 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.924+0000 7f4e0921c640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e04095fb0 0x7f4e04096380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.924+0000 7f4e0921c640 10 monclient: get_auth_request con 0x7f4e04095fb0 auth_method 0 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.924+0000 7f4e0921c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.924+0000 7f4e0921c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.924+0000 7f4e0921c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.924+0000 7f4e0921c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:51.941 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.924+0000 7f4e0921c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:51.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.924+0000 7f4e0921c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:51.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.925+0000 7f4e0921c640 10 monclient(hunting): handle_auth_done global_id 54690 payload 824 2024-09-15T04:02:51.942 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.925+0000 7f4e0921c640 1 -- 172.21.15.110:0/482496775 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4e040647e0 con 0x7f4e04095fb0 2024-09-15T04:02:51.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e0921c640 1 -- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e04095fb0 msgr2=0x7f4e04096380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:51.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e0921c640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e04095fb0 0x7f4e04096380 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4df400e210 tx=0x7f4df403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:51.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e017fa640 1 -- 172.21.15.110:0/482496775 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4043d90 con 0x7f4e04095fb0 2024-09-15T04:02:51.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:51.943 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:51.944 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e017fa640 10 monclient: dump: 2024-09-15T04:02:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:51.945 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e017fa640 1 -- 172.21.15.110:0/482496775 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7f4df404f3e0 con 0x7f4e04095fb0 2024-09-15T04:02:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e017fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:02:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e017fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:02:51.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e017fa640 1 -- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e04095fb0 msgr2=0x7f4e04096380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e017fa640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e04095fb0 0x7f4e04096380 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4df400e210 tx=0x7f4df403f000 comp rx=0 tx=0).stop 2024-09-15T04:02:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e017fa640 1 -- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e04095fb0 msgr2=0x7f4e04096380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:02:51.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e017fa640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e04095fb0 0x7f4e04096380 secure :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4df400e210 tx=0x7f4df403f000 comp rx=0 tx=0).stop 2024-09-15T04:02:51.946 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:02:51.946 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:02:51.946 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:02:51.946 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "leader", 2024-09-15T04:02:51.946 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 28, 2024-09-15T04:02:51.946 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:02:51.946 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:02:51.947 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:02:51.947 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:02:51.947 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:51.947 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 1, 2024-09-15T04:02:51.947 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:02:51.947 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:02:51.947 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:02:51.947 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:51.947 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:51.947 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:51.947 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:51.947 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:51.947 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:51.947 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:02:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:02:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:51.948 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:02:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:02:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:02:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:02:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:02:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:02:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:02:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:02:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:02:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:02:51.949 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:02:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:02:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:02:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:02:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:02:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:02:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:02:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:51.950 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:02:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:02:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:02:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:02:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:02:51.951 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:02:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:02:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:02:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:51.952 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:02:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:02:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:02:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:51.953 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:51.954 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:02:51.955 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T04:02:51.956 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:02:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:02:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 3 2024-09-15T04:02:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:51.957 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:02:51.958 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:02:51.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e0b480640 1 -- 172.21.15.110:0/482496775 >> v2:172.21.15.177:6812/28201 conn(0x7f4df0038730 msgr2=0x7f4df003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:51.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e0b480640 1 --2- 172.21.15.110:0/482496775 >> v2:172.21.15.177:6812/28201 conn(0x7f4df0038730 0x7f4df003abe0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4df8001990 tx=0x7f4df8009040 comp rx=0 tx=0).stop 2024-09-15T04:02:51.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e0b480640 10 monclient: shutdown 2024-09-15T04:02:51.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e0b480640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:51.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e0b480640 1 -- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e04099c40 msgr2=0x7f4e0409a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:51.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e0b480640 1 --2- 172.21.15.110:0/482496775 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4e04099c40 0x7f4e0409a0b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4e04132700 tx=0x7f4dfc009040 comp rx=0 tx=0).stop 2024-09-15T04:02:51.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e09a1d640 1 -- 172.21.15.110:0/482496775 reap_dead start 2024-09-15T04:02:51.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.927+0000 7f4e09a1d640 1 -- 172.21.15.110:0/482496775 reap_dead start 2024-09-15T04:02:51.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.928+0000 7f4e0b480640 1 -- 172.21.15.110:0/482496775 shutdown_connections 2024-09-15T04:02:51.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.928+0000 7f4e0b480640 1 -- 172.21.15.110:0/482496775 >> 172.21.15.110:0/482496775 conn(0x7f4e0406e860 msgr2=0x7f4e0409d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:51.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.928+0000 7f4e0b480640 1 -- 172.21.15.110:0/482496775 shutdown_connections 2024-09-15T04:02:51.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:51.928+0000 7f4e0b480640 1 -- 172.21.15.110:0/482496775 wait complete. 2024-09-15T04:02:51.960 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:02:51.960 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T04:02:51.960 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T04:02:52.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.052+0000 7fe60f460640 10 monclient: get_monmap_and_config 2024-09-15T04:02:52.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.052+0000 7fe60f460640 10 monclient: build_initial_monmap 2024-09-15T04:02:52.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.052+0000 7fe60f460640 10 monclient: monmap: 2024-09-15T04:02:52.054 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:52.054 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:52.054 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:02:52.053796+0000 2024-09-15T04:02:52.054 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:02:52.053796+0000 2024-09-15T04:02:52.054 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:52.054 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:52.054 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:02:52.054 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:02:52.054 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:02:52.054 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:02:52.054 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:02:52.055 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:52.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.053+0000 7fe60f460640 1 Processor -- start 2024-09-15T04:02:52.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.053+0000 7fe60f460640 1 -- start start 2024-09-15T04:02:52.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.053+0000 7fe60f460640 10 monclient: init 2024-09-15T04:02:52.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.053+0000 7fe60f460640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:52.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60f460640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-15T04:02:52.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60f460640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe60812f670 addr v1:172.21.15.110:6789/0 2024-09-15T04:02:52.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60f460640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe608063de0 addr v1:172.21.15.177:6789/0 2024-09-15T04:02:52.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60f460640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe6081359a0 0x7fe608137d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:52.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60f460640 10 monclient(hunting): picked mon.noname-a con 0x7fe6081359a0 addr v2:172.21.15.110:3300/0 2024-09-15T04:02:52.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60f460640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60806cf90 con 0x7fe60812f670 2024-09-15T04:02:52.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60f460640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6080641e0 con 0x7fe608063de0 2024-09-15T04:02:52.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60f460640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:52.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60f460640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe60806b800 con 0x7fe6081359a0 2024-09-15T04:02:52.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60f460640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:52.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60f460640 10 monclient(hunting): authenticate will time out at 995.698081s 2024-09-15T04:02:52.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60d9fd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe6081359a0 0x7fe608137d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:52.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60d9fd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe6081359a0 0x7fe608137d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40362/0 (socket says 172.21.15.110:40362) 2024-09-15T04:02:52.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60d9fd640 1 -- 172.21.15.110:0/3924547476 learned_addr learned my addr 172.21.15.110:0/3924547476 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:52.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60d9fd640 10 monclient(hunting): get_auth_request con 0x7fe6081359a0 auth_method 0 2024-09-15T04:02:52.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60d9fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:52.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60d9fd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:52.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60d9fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:52.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60d9fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:52.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60d9fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:52.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.054+0000 7fe60d9fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:52.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.056+0000 7fe60d1fc640 1 -- 172.21.15.110:0/3924547476 >> v1:172.21.15.110:6789/0 conn(0x7fe60812f670 legacy=0x7fe60806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:52.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.056+0000 7fe60d9fd640 10 monclient(hunting): handle_auth_done global_id 54693 payload 274 2024-09-15T04:02:52.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.056+0000 7fe60d9fd640 1 -- 172.21.15.110:0/3924547476 >> v1:172.21.15.177:6789/0 conn(0x7fe608063de0 legacy=0x7fe60812fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:52.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.056+0000 7fe60d9fd640 1 -- 172.21.15.110:0/3924547476 >> v1:172.21.15.110:6789/0 conn(0x7fe60812f670 legacy=0x7fe60806c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:52.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.056+0000 7fe60d9fd640 10 monclient: _finish_hunting 0 2024-09-15T04:02:52.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.056+0000 7fe60d9fd640 1 monclient: found mon.noname-a 2024-09-15T04:02:52.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.056+0000 7fe60d9fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:52.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.056+0000 7fe60d9fd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:02:52.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.056+0000 7fe60d9fd640 1 -- 172.21.15.110:0/3924547476 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6080a0800 con 0x7fe6081359a0 2024-09-15T04:02:52.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe60d9fd640 1 -- 172.21.15.110:0/3924547476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6081359a0 msgr2=0x7fe608137d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:52.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe60d9fd640 1 --2- 172.21.15.110:0/3924547476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6081359a0 0x7fe608137d80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe600009fa0 tx=0x7fe600009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9449a47e8cae02a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:52.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe5fffff640 1 -- 172.21.15.110:0/3924547476 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe600007610 con 0x7fe6081359a0 2024-09-15T04:02:52.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe5fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:52.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe5fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe5fffff640 10 monclient: dump: 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe5fffff640 10 monclient: _finish_auth 0 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe5fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe5fffff640 1 -- 172.21.15.110:0/3924547476 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60002d050 con 0x7fe6081359a0 2024-09-15T04:02:52.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe5fffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe5fffff640 1 -- 172.21.15.110:0/3924547476 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6000079a0 con 0x7fe6081359a0 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe60f460640 5 monclient: authenticate success, global_id 54693 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe60f460640 10 monclient: get_monmap_and_config success 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe60f460640 4 set_mon_vals no callback set 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe5fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe5fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.059+0000 7fe5fffff640 10 monclient: dump: 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:52.063 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:52.064 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:52.064 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:52.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.060+0000 7fe60f460640 10 monclient: shutdown 2024-09-15T04:02:52.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.060+0000 7fe60f460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:52.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.060+0000 7fe60f460640 1 -- 172.21.15.110:0/3924547476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6081359a0 msgr2=0x7fe608137d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:52.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.060+0000 7fe60f460640 1 --2- 172.21.15.110:0/3924547476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6081359a0 0x7fe608137d80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe600009fa0 tx=0x7fe600009840 comp rx=0 tx=0).stop 2024-09-15T04:02:52.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.060+0000 7fe60f460640 1 -- 172.21.15.110:0/3924547476 shutdown_connections 2024-09-15T04:02:52.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.060+0000 7fe60f460640 1 --2- 172.21.15.110:0/3924547476 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6081359a0 0x7fe608137d80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:52.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.060+0000 7fe60f460640 1 -- 172.21.15.110:0/3924547476 >> 172.21.15.110:0/3924547476 conn(0x7fe60809e570 msgr2=0x7fe60809e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:52.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.060+0000 7fe60f460640 1 -- 172.21.15.110:0/3924547476 shutdown_connections 2024-09-15T04:02:52.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.060+0000 7fe60f460640 1 -- 172.21.15.110:0/3924547476 wait complete. 2024-09-15T04:02:52.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.060+0000 7fe60f460640 10 monclient: build_initial_monmap 2024-09-15T04:02:52.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.060+0000 7fe60f460640 10 monclient: monmap: 2024-09-15T04:02:52.064 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:52.065 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:02:52.065 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:02:52.065 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:02:52.065 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:52.065 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:52.065 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:02:52.065 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:02:52.065 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:02:52.065 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:52.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 1 Processor -- start 2024-09-15T04:02:52.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 1 -- start start 2024-09-15T04:02:52.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 10 monclient: init 2024-09-15T04:02:52.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:52.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T04:02:52.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe608063de0 0x7fe608134af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:52.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 10 monclient(hunting): picked mon.noname-b con 0x7fe608063de0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:02:52.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812f670 0x7fe60812d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:52.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 10 monclient(hunting): picked mon.noname-c con 0x7fe60812f670 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:52.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6081359a0 0x7fe60812df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:52.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 10 monclient(hunting): picked mon.noname-a con 0x7fe6081359a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:52.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:52.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080637e0 con 0x7fe6081359a0 2024-09-15T04:02:52.070 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:52.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080a4200 con 0x7fe608063de0 2024-09-15T04:02:52.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:52.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080a44e0 con 0x7fe60812f670 2024-09-15T04:02:52.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:52.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60f460640 10 monclient(hunting): authenticate will time out at 995.705689s 2024-09-15T04:02:52.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60d9fd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6081359a0 0x7fe60812df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:52.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60c9fb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812f670 0x7fe60812d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:52.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60d9fd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6081359a0 0x7fe60812df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40364/0 (socket says 172.21.15.110:40364) 2024-09-15T04:02:52.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60d9fd640 1 -- 172.21.15.110:0/2159124229 learned_addr learned my addr 172.21.15.110:0/2159124229 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:52.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60c9fb640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812f670 0x7fe60812d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52170/0 (socket says 172.21.15.110:52170) 2024-09-15T04:02:52.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60d9fd640 10 monclient(hunting): get_auth_request con 0x7fe6081359a0 auth_method 0 2024-09-15T04:02:52.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60d9fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:52.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60d9fd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60d9fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60c9fb640 10 monclient(hunting): get_auth_request con 0x7fe60812f670 auth_method 0 2024-09-15T04:02:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60c9fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60c9fb640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60c9fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60d9fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60d9fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60d9fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60c9fb640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812f670 msgr2=0x7fe60812d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T04:02:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60c9fb640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812f670 msgr2=0x7fe60812d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T04:02:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60c9fb640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812f670 0x7fe60812d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:02:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60c9fb640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812f670 0x7fe60812d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:02:52.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.061+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe608063de0 0x7fe608134af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.062+0000 7fe60d1fc640 10 monclient(hunting): get_auth_request con 0x7fe608063de0 auth_method 0 2024-09-15T04:02:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.062+0000 7fe60d1fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.062+0000 7fe60d1fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.062+0000 7fe60d1fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.062+0000 7fe60d9fd640 10 monclient(hunting): handle_auth_done global_id 54696 payload 824 2024-09-15T04:02:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.062+0000 7fe60d9fd640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812f670 msgr2=0x7fe60812d9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:02:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.062+0000 7fe60d9fd640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812f670 0x7fe60812d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:52.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.062+0000 7fe60d9fd640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe608063de0 msgr2=0x7fe608134af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.062+0000 7fe60d9fd640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe608063de0 0x7fe608134af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.062+0000 7fe60d9fd640 10 monclient: _finish_hunting 0 2024-09-15T04:02:52.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.062+0000 7fe60d9fd640 1 monclient: found mon.noname-a 2024-09-15T04:02:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.062+0000 7fe60d9fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.062+0000 7fe60d9fd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:02:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.062+0000 7fe60d9fd640 1 -- 172.21.15.110:0/2159124229 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe600009ae0 con 0x7fe6081359a0 2024-09-15T04:02:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe60d9fd640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6081359a0 msgr2=0x7fe60812df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe60d9fd640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6081359a0 0x7fe60812df00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe600009fa0 tx=0x7fe600036000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe5fd7fa640 1 -- 172.21.15.110:0/2159124229 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe600032ce0 con 0x7fe6081359a0 2024-09-15T04:02:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe5fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe5fd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:02:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe5fd7fa640 10 monclient: dump: 2024-09-15T04:02:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:52.077 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe5fd7fa640 10 monclient: _finish_auth 0 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe5fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe5fd7fa640 1 -- 172.21.15.110:0/2159124229 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe600032e40 con 0x7fe6081359a0 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe60f460640 5 monclient: authenticate success, global_id 54696 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe60f460640 10 monclient: _renew_subs 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe60f460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe60f460640 1 -- 172.21.15.110:0/2159124229 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60809b670 con 0x7fe6081359a0 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe5fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe5fd7fa640 1 -- 172.21.15.110:0/2159124229 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60003b640 con 0x7fe6081359a0 2024-09-15T04:02:52.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe5fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe5fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe5fd7fa640 10 monclient: dump: 2024-09-15T04:02:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:52.079 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe60f460640 10 monclient: _renew_subs 2024-09-15T04:02:52.079 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe60f460640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe60f460640 1 -- 172.21.15.110:0/2159124229 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60812ef20 con 0x7fe6081359a0 2024-09-15T04:02:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.064+0000 7fe5ff7fe640 4 set_mon_vals no callback set 2024-09-15T04:02:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60f460640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe5fd7fa640 1 -- 172.21.15.110:0/2159124229 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fe600038050 con 0x7fe6081359a0 2024-09-15T04:02:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60f460640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60f460640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60f460640 1 -- 172.21.15.110:0/2159124229 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe608068c10 con 0x7fe60812e440 2024-09-15T04:02:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60d1fc640 10 monclient: get_auth_request con 0x7fe60812e440 auth_method 0 2024-09-15T04:02:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60d1fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60d1fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60d1fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:52.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60d1fc640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 msgr2=0x7fe60812a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T04:02:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60d1fc640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 msgr2=0x7fe60812a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T04:02:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:02:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:02:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe5fd7fa640 1 --2- 172.21.15.110:0/2159124229 >> v2:172.21.15.177:6812/28201 conn(0x7fe5ec038700 0x7fe5ec03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe5fd7fa640 1 -- 172.21.15.110:0/2159124229 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fe60006c370 con 0x7fe6081359a0 2024-09-15T04:02:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60c9fb640 1 --2- 172.21.15.110:0/2159124229 >> v2:172.21.15.177:6812/28201 conn(0x7fe5ec038700 0x7fe5ec03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.065+0000 7fe60c9fb640 10 monclient: get_auth_request con 0x7fe5ec038700 auth_method 0 2024-09-15T04:02:52.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.068+0000 7fe60c9fb640 1 --2- 172.21.15.110:0/2159124229 >> v2:172.21.15.177:6812/28201 conn(0x7fe5ec038700 0x7fe5ec03abb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe608135250 tx=0x7fe5f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:52.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.266+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:52.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.266+0000 7fe60d1fc640 10 monclient: get_auth_request con 0x7fe60812e440 auth_method 0 2024-09-15T04:02:52.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.266+0000 7fe60d1fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:52.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.266+0000 7fe60d1fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:52.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.266+0000 7fe60d1fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:02:52.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.266+0000 7fe60d1fc640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 msgr2=0x7fe60812a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T04:02:52.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.266+0000 7fe60d1fc640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 msgr2=0x7fe60812a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T04:02:52.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.266+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:02:52.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.266+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T04:02:52.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.667+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:52.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.667+0000 7fe60d1fc640 10 monclient: get_auth_request con 0x7fe60812e440 auth_method 0 2024-09-15T04:02:52.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.667+0000 7fe60d1fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:52.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.667+0000 7fe60d1fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:52.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.667+0000 7fe60d1fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:02:52.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.667+0000 7fe60d1fc640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 msgr2=0x7fe60812a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T04:02:52.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.667+0000 7fe60d1fc640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 msgr2=0x7fe60812a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T04:02:52.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.667+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:02:52.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:52.667+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T04:02:53.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:53.468+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:53.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:53.468+0000 7fe60d1fc640 10 monclient: get_auth_request con 0x7fe60812e440 auth_method 0 2024-09-15T04:02:53.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:53.468+0000 7fe60d1fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:53.469 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:53.468+0000 7fe60d1fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:53.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:53.468+0000 7fe60d1fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:02:53.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:53.468+0000 7fe60d1fc640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 msgr2=0x7fe60812a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T04:02:53.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:53.468+0000 7fe60d1fc640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 msgr2=0x7fe60812a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T04:02:53.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:53.468+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:02:53.470 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:53.468+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T04:02:53.648 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32558]: starting. 2024-09-15T04:02:53.648 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32558]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32557 2024-09-15T04:02:53.648 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32558]: created object 0... 2024-09-15T04:02:53.648 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32558]: created object 25... 2024-09-15T04:02:53.648 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32558]: created object 49... 2024-09-15T04:02:53.648 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32558]: finishing. 2024-09-15T04:02:53.649 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32558]: shutting down. 2024-09-15T04:02:54.573 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32559]: starting. 2024-09-15T04:02:54.574 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32559]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32557 2024-09-15T04:02:54.574 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32559]: shutting down. 2024-09-15T04:02:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.061+0000 7fe5f3fff640 10 monclient: tick 2024-09-15T04:02:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.061+0000 7fe5f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:55.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.061+0000 7fe5f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:55.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.070+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.070+0000 7fe60d1fc640 10 monclient: get_auth_request con 0x7fe60812e440 auth_method 0 2024-09-15T04:02:55.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.070+0000 7fe60d1fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.070+0000 7fe60d1fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.070+0000 7fe60d1fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:02:55.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.070+0000 7fe60d1fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:55.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.070+0000 7fe60d1fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:55.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.070+0000 7fe60d1fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:55.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.070+0000 7fe60d1fc640 10 monclient(hunting): handle_auth_done global_id 64100 payload 824 2024-09-15T04:02:55.072 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.070+0000 7fe60d1fc640 1 -- 172.21.15.110:0/2159124229 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6080647e0 con 0x7fe60812e440 2024-09-15T04:02:55.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.073+0000 7fe60d1fc640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 msgr2=0x7fe60812a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:55.073 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.073+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe6040146a0 tx=0x7fe604009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:55.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.073+0000 7fe5fd7fa640 1 -- 172.21.15.110:0/2159124229 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6040075c0 con 0x7fe60812e440 2024-09-15T04:02:55.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.073+0000 7fe5fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:55.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.073+0000 7fe5fd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:02:55.074 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.073+0000 7fe5fd7fa640 10 monclient: dump: 2024-09-15T04:02:55.074 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:55.074 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.074 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.074 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.074 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:55.074 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.074 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:55.074 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:55.074 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:55.075 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.073+0000 7fe5fd7fa640 1 -- 172.21.15.110:0/2159124229 <== mon.2 v2:172.21.15.110:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe604005780 con 0x7fe60812e440 2024-09-15T04:02:55.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.073+0000 7fe5fd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:55.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.073+0000 7fe5fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:02:55.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.073+0000 7fe5fd7fa640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 msgr2=0x7fe60812a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.075 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.073+0000 7fe5fd7fa640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe6040146a0 tx=0x7fe604009040 comp rx=0 tx=0).stop 2024-09-15T04:02:55.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.073+0000 7fe5fd7fa640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 msgr2=0x7fe60812a7c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:02:55.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.073+0000 7fe5fd7fa640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e440 0x7fe60812a7c0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe6040146a0 tx=0x7fe604009040 comp rx=0 tx=0).stop 2024-09-15T04:02:55.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.080+0000 7fe60f460640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:02:55.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.080+0000 7fe60f460640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e810 0x7fe60812ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.080+0000 7fe60f460640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.080+0000 7fe60f460640 1 -- 172.21.15.110:0/2159124229 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080a3e30 con 0x7fe60812e810 2024-09-15T04:02:55.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.080+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e810 0x7fe60812ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.080+0000 7fe60d1fc640 10 monclient: get_auth_request con 0x7fe60812e810 auth_method 0 2024-09-15T04:02:55.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.080+0000 7fe60d1fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.080+0000 7fe60d1fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.080+0000 7fe60d1fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:55.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.080+0000 7fe60d1fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:55.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.080+0000 7fe60d1fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:55.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.080+0000 7fe60d1fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:55.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.080+0000 7fe60d1fc640 10 monclient(hunting): handle_auth_done global_id 64103 payload 824 2024-09-15T04:02:55.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.080+0000 7fe60d1fc640 1 -- 172.21.15.110:0/2159124229 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe6080691e0 con 0x7fe60812e810 2024-09-15T04:02:55.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.083+0000 7fe60d1fc640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e810 msgr2=0x7fe60812ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:55.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.083+0000 7fe60d1fc640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e810 0x7fe60812ad00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe60401c000 tx=0x7fe60401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:55.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.083+0000 7fe5fd7fa640 1 -- 172.21.15.110:0/2159124229 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604012e60 con 0x7fe60812e810 2024-09-15T04:02:55.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.083+0000 7fe5fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:55.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.083+0000 7fe5fd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:02:55.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.083+0000 7fe5fd7fa640 10 monclient: dump: 2024-09-15T04:02:55.085 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:55.085 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.085 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.085 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.085 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:55.085 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.085 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:55.085 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:55.085 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:55.085 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.083+0000 7fe5fd7fa640 1 -- 172.21.15.110:0/2159124229 <== mon.2 v2:172.21.15.110:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7fe60401f870 con 0x7fe60812e810 2024-09-15T04:02:55.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.083+0000 7fe5fd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:02:55.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.083+0000 7fe5fd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:02:55.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.083+0000 7fe5fd7fa640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e810 msgr2=0x7fe60812ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.083+0000 7fe5fd7fa640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e810 0x7fe60812ad00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe60401c000 tx=0x7fe60401f000 comp rx=0 tx=0).stop 2024-09-15T04:02:55.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.083+0000 7fe5fd7fa640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e810 msgr2=0x7fe60812ad00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:02:55.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.083+0000 7fe5fd7fa640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe60812e810 0x7fe60812ad00 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe60401c000 tx=0x7fe60401f000 comp rx=0 tx=0).stop 2024-09-15T04:02:55.086 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:02:55.086 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:02:55.086 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:02:55.086 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T04:02:55.086 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 28, 2024-09-15T04:02:55.087 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:02:55.087 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:02:55.087 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:02:55.087 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:02:55.087 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:55.087 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 4, 2024-09-15T04:02:55.087 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:02:55.087 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:02:55.087 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:02:55.087 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:55.087 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:55.087 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:55.087 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:55.088 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:55.088 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:55.088 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:55.088 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:55.088 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:55.088 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:55.088 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:02:55.088 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:02:55.088 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:55.088 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:55.088 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:55.088 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:55.088 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:55.089 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:55.089 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:55.089 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:55.089 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:55.089 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:55.089 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.089 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:02:55.089 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:02:55.089 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:02:55.089 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:02:55.089 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:02:55.089 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:02:55.090 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:02:55.090 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:02:55.090 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:02:55.090 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:02:55.090 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:02:55.090 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:02:55.090 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:02:55.090 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:02:55.090 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:02:55.090 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:02:55.090 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:02:55.090 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:55.090 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:55.090 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:55.091 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:55.091 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:55.091 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:55.091 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:55.091 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:55.091 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:55.091 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:55.091 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:02:55.091 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.091 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:02:55.091 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.091 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:02:55.091 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:02:55.092 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:55.092 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:55.092 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.092 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:55.092 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:02:55.092 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:55.092 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.092 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.092 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:55.092 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:02:55.092 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:55.092 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:55.092 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:55.093 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.093 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:02:55.093 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:02:55.093 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:55.093 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:55.093 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:55.093 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.093 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.093 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:02:55.093 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:02:55.093 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:55.093 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:55.093 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.094 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:55.094 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:02:55.094 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:55.094 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.094 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.094 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:55.094 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:02:55.094 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:55.094 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:55.094 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:55.094 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.094 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:02:55.094 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:02:55.095 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:55.095 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:55.095 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:55.095 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.095 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.095 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:02:55.095 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:02:55.095 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:55.095 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:55.095 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.095 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:55.095 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:02:55.095 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:55.095 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.096 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.096 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:55.096 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:02:55.096 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:55.096 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:55.096 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:55.096 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.096 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:02:55.096 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:02:55.096 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:55.096 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:55.096 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:55.096 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:55.097 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:55.097 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.097 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:02:55.097 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:02:55.097 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.097 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:55.097 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:55.097 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:02:55.097 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:55.097 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:55.097 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:02:55.097 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.097 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:55.097 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:55.098 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:02:55.098 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:55.098 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:55.098 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.098 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:02:55.098 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:02:55.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.084+0000 7fe60f460640 1 -- 172.21.15.110:0/2159124229 >> v2:172.21.15.177:6812/28201 conn(0x7fe5ec038700 msgr2=0x7fe5ec03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.084+0000 7fe60f460640 1 --2- 172.21.15.110:0/2159124229 >> v2:172.21.15.177:6812/28201 conn(0x7fe5ec038700 0x7fe5ec03abb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe608135250 tx=0x7fe5f8009040 comp rx=0 tx=0).stop 2024-09-15T04:02:55.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.084+0000 7fe60f460640 10 monclient: shutdown 2024-09-15T04:02:55.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.084+0000 7fe60f460640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:55.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.084+0000 7fe60f460640 1 -- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6081359a0 msgr2=0x7fe60812df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.099 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.084+0000 7fe60f460640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6081359a0 0x7fe60812df00 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe600009fa0 tx=0x7fe600036000 comp rx=0 tx=0).stop 2024-09-15T04:02:55.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.084+0000 7fe60d9fd640 1 -- 172.21.15.110:0/2159124229 reap_dead start 2024-09-15T04:02:55.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.084+0000 7fe60f460640 1 -- 172.21.15.110:0/2159124229 shutdown_connections 2024-09-15T04:02:55.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.084+0000 7fe60f460640 1 --2- 172.21.15.110:0/2159124229 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe6081359a0 0x7fe60812df00 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.084+0000 7fe60f460640 1 -- 172.21.15.110:0/2159124229 >> 172.21.15.110:0/2159124229 conn(0x7fe60809e570 msgr2=0x7fe608129450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:55.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.084+0000 7fe60f460640 1 -- 172.21.15.110:0/2159124229 shutdown_connections 2024-09-15T04:02:55.100 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.084+0000 7fe60f460640 1 -- 172.21.15.110:0/2159124229 wait complete. 2024-09-15T04:02:55.100 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:02:55.100 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T04:02:55.101 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T04:02:55.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.202+0000 7f77078af640 10 monclient: get_monmap_and_config 2024-09-15T04:02:55.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.202+0000 7f77078af640 10 monclient: build_initial_monmap 2024-09-15T04:02:55.203 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.202+0000 7f77078af640 10 monclient: monmap: 2024-09-15T04:02:55.203 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:55.203 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.203 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:02:55.203039+0000 2024-09-15T04:02:55.203 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:02:55.203039+0000 2024-09-15T04:02:55.203 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:55.204 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.204 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:02:55.204 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:02:55.204 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:02:55.204 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:02:55.204 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:02:55.204 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.202+0000 7f77078af640 1 Processor -- start 2024-09-15T04:02:55.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.202+0000 7f77078af640 1 -- start start 2024-09-15T04:02:55.204 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.202+0000 7f77078af640 10 monclient: init 2024-09-15T04:02:55.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f77078af640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:55.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f77078af640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-15T04:02:55.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f77078af640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f77000a1a10 0x7f7700131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f77078af640 10 monclient(hunting): picked mon.noname-c con 0x7f77000a1a10 addr v2:172.21.15.177:3300/0 2024-09-15T04:02:55.209 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f77078af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f770009e770 addr v1:172.21.15.110:6789/0 2024-09-15T04:02:55.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f77078af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77001369e0 addr v1:172.21.15.177:6789/0 2024-09-15T04:02:55.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f77078af640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f770009f2c0 con 0x7f770009e770 2024-09-15T04:02:55.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f77078af640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77000a0850 con 0x7f77001369e0 2024-09-15T04:02:55.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f77078af640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f77078af640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f770009c9a0 con 0x7f77000a1a10 2024-09-15T04:02:55.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f77078af640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:55.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f77078af640 10 monclient(hunting): authenticate will time out at 998.847478s 2024-09-15T04:02:55.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f7704e4a640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f770009e770 0x7f770009ebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53778/0 (socket says 172.21.15.110:53778) 2024-09-15T04:02:55.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f7704e4a640 1 -- 172.21.15.110:0/2495296647 learned_addr learned my addr 172.21.15.110:0/2495296647 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:02:55.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f770564b640 1 --2- 172.21.15.110:0/2495296647 >> v2:172.21.15.177:3300/0 conn(0x7f77000a1a10 0x7f7700131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f770564b640 10 monclient(hunting): get_auth_request con 0x7f77000a1a10 auth_method 0 2024-09-15T04:02:55.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f770564b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f770564b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.210 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f770564b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:55.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:55.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:55.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.203+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:55.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.205+0000 7f7704e4a640 1 -- 172.21.15.110:0/2495296647 >> v1:172.21.15.110:6789/0 conn(0x7f770009e770 legacy=0x7f770009ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:55.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.205+0000 7f76f7fff640 1 -- 172.21.15.110:0/2495296647 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554563492 0 0) 0x7f770009f2c0 con 0x7f770009e770 2024-09-15T04:02:55.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.205+0000 7f76f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.205+0000 7f76f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:55.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.205+0000 7f76f7fff640 10 monclient(hunting): my global_id is 54720 2024-09-15T04:02:55.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.205+0000 7f76f7fff640 1 -- 172.21.15.110:0/2495296647 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76ec0034e0 con 0x7f770009e770 2024-09-15T04:02:55.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7705e4c640 1 -- 172.21.15.110:0/2495296647 >> v1:172.21.15.177:6789/0 conn(0x7f77001369e0 legacy=0x7f7700138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:55.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 -- 172.21.15.110:0/2495296647 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 163827738 0 0) 0x7f76ec0034e0 con 0x7f770009e770 2024-09-15T04:02:55.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 -- 172.21.15.110:0/2495296647 >> v2:172.21.15.177:3300/0 conn(0x7f77000a1a10 msgr2=0x7f7700131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:55.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 --2- 172.21.15.110:0/2495296647 >> v2:172.21.15.177:3300/0 conn(0x7f77000a1a10 0x7f7700131b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.211 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 -- 172.21.15.110:0/2495296647 >> v1:172.21.15.177:6789/0 conn(0x7f77001369e0 legacy=0x7f7700138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient: _finish_hunting 0 2024-09-15T04:02:55.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 monclient: found mon.noname-a-legacy 2024-09-15T04:02:55.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:55.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.110:6789/0 2024-09-15T04:02:55.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 -- 172.21.15.110:0/2495296647 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770009cc80 con 0x7f770009e770 2024-09-15T04:02:55.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient: _finish_auth 0 2024-09-15T04:02:55.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:55.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 -- 172.21.15.110:0/2495296647 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3165669920 0 0) 0x7f76f00017e0 con 0x7f770009e770 2024-09-15T04:02:55.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f77078af640 5 monclient: authenticate success, global_id 54720 2024-09-15T04:02:55.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f77078af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:02:55.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:55.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T04:02:55.212 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient: dump: 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] but i'm connected to v1:172.21.15.110:6789/0, reconnecting 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 -- 172.21.15.110:0/2495296647 >> v1:172.21.15.110:6789/0 conn(0x7f770009e770 legacy=0x7f770009ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T04:02:55.213 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 --2- 172.21.15.110:0/2495296647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76ec003a50 0x7f76ec003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient(hunting): picked mon.a con 0x7f76ec003a50 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:55.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 --2- 172.21.15.110:0/2495296647 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76ec004780 0x7f76ec004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient(hunting): picked mon.c con 0x7f76ec004780 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:55.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 --2- 172.21.15.110:0/2495296647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76ec005230 0x7f76ec00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient(hunting): picked mon.b con 0x7f76ec005230 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:02:55.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 -- 172.21.15.110:0/2495296647 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770009c9a0 con 0x7f76ec003a50 2024-09-15T04:02:55.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.214 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 -- 172.21.15.110:0/2495296647 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec00c1f0 con 0x7f76ec005230 2024-09-15T04:02:55.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 1 -- 172.21.15.110:0/2495296647 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec00c3b0 con 0x7f76ec004780 2024-09-15T04:02:55.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:55.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f76f7fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T04:02:55.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f77078af640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T04:02:55.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 1 --2- 172.21.15.110:0/2495296647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76ec003a50 0x7f76ec003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 10 monclient(hunting): get_auth_request con 0x7f76ec003a50 auth_method 0 2024-09-15T04:02:55.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:02:55.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7704e4a640 1 --2- 172.21.15.110:0/2495296647 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76ec004780 0x7f76ec004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7704e4a640 10 monclient(hunting): get_auth_request con 0x7f76ec004780 auth_method 0 2024-09-15T04:02:55.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7704e4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.215 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7704e4a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7704e4a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:02:55.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:55.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:55.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:02:55.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7705e4c640 1 --2- 172.21.15.110:0/2495296647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76ec005230 0x7f76ec00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7705e4c640 10 monclient(hunting): get_auth_request con 0x7f76ec005230 auth_method 0 2024-09-15T04:02:55.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7705e4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7705e4c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7705e4c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:02:55.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7704e4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:55.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7704e4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:55.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7704e4a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:02:55.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 10 monclient(hunting): handle_auth_done global_id 54720 payload 293 2024-09-15T04:02:55.216 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 1 -- 172.21.15.110:0/2495296647 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76ec004780 msgr2=0x7f76ec004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:55.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 1 --2- 172.21.15.110:0/2495296647 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76ec004780 0x7f76ec004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 1 -- 172.21.15.110:0/2495296647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76ec005230 msgr2=0x7f76ec00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:55.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 1 --2- 172.21.15.110:0/2495296647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76ec005230 0x7f76ec00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 10 monclient: _finish_hunting 0 2024-09-15T04:02:55.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 1 monclient: found mon.a 2024-09-15T04:02:55.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:55.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:55.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 1 -- 172.21.15.110:0/2495296647 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec0034e0 con 0x7f76ec003a50 2024-09-15T04:02:55.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 10 monclient: _finish_auth 0 2024-09-15T04:02:55.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f770564b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:55.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7705e4c640 1 --2- 172.21.15.110:0/2495296647 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f76ec005230 0x7f76ec00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T04:02:55.217 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.206+0000 7f7705e4c640 1 -- 172.21.15.110:0/2495296647 reap_dead start 2024-09-15T04:02:55.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f770564b640 1 -- 172.21.15.110:0/2495296647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76ec003a50 msgr2=0x7f76ec003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:55.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f770564b640 1 --2- 172.21.15.110:0/2495296647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76ec003a50 0x7f76ec003e60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009b70 tx=0x7f76fc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c13dc4c477b3f41 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:55.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f76f7fff640 1 -- 172.21.15.110:0/2495296647 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc02f510 con 0x7f76ec003a50 2024-09-15T04:02:55.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f76f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:55.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f76f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:55.224 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f76f7fff640 10 monclient: dump: 2024-09-15T04:02:55.224 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:55.225 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.225 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.225 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.225 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:55.225 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.225 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:55.225 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:55.225 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:55.225 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f77078af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:02:55.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f76f7fff640 1 -- 172.21.15.110:0/2495296647 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76fc02d050 con 0x7f76ec003a50 2024-09-15T04:02:55.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f76f7fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:55.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f77078af640 10 monclient: get_monmap_and_config success 2024-09-15T04:02:55.225 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f76f7fff640 1 -- 172.21.15.110:0/2495296647 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc02f8a0 con 0x7f76ec003a50 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f77078af640 4 set_mon_vals no callback set 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f76f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f76f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.209+0000 7f76f7fff640 10 monclient: dump: 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.226 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.210+0000 7f77078af640 10 monclient: shutdown 2024-09-15T04:02:55.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.210+0000 7f77078af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:55.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.210+0000 7f77078af640 1 -- 172.21.15.110:0/2495296647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76ec003a50 msgr2=0x7f76ec003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.210+0000 7f77078af640 1 --2- 172.21.15.110:0/2495296647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76ec003a50 0x7f76ec003e60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009b70 tx=0x7f76fc0090f0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.210+0000 7f77078af640 1 -- 172.21.15.110:0/2495296647 shutdown_connections 2024-09-15T04:02:55.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.210+0000 7f77078af640 1 --2- 172.21.15.110:0/2495296647 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76ec003a50 0x7f76ec003e60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.210+0000 7f77078af640 1 -- 172.21.15.110:0/2495296647 >> 172.21.15.110:0/2495296647 conn(0x7f770006e860 msgr2=0x7f770006ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:55.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.210+0000 7f77078af640 1 -- 172.21.15.110:0/2495296647 shutdown_connections 2024-09-15T04:02:55.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.210+0000 7f77078af640 1 -- 172.21.15.110:0/2495296647 wait complete. 2024-09-15T04:02:55.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.210+0000 7f77078af640 10 monclient: build_initial_monmap 2024-09-15T04:02:55.227 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.210+0000 7f77078af640 10 monclient: monmap: 2024-09-15T04:02:55.227 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:55.227 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:02:55.227 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:02:55.227 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:02:55.228 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:55.228 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.228 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:02:55.228 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:02:55.228 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:02:55.228 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 1 Processor -- start 2024-09-15T04:02:55.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 1 -- start start 2024-09-15T04:02:55.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 10 monclient: init 2024-09-15T04:02:55.228 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76ec003a50 0x7f77001305b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 10 monclient(hunting): picked mon.noname-c con 0x7f76ec003a50 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7700130af0 0x7f7700099700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 10 monclient(hunting): picked mon.noname-a con 0x7f7700130af0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7700099c40 0x7f770009a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 10 monclient(hunting): picked mon.noname-b con 0x7f7700099c40 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7700130ee0 con 0x7f7700130af0 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77000a11c0 con 0x7f7700099c40 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77000a46f0 con 0x7f76ec003a50 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:55.229 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f77078af640 10 monclient(hunting): authenticate will time out at 998.855720s 2024-09-15T04:02:55.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f770564b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76ec003a50 0x7f77001305b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7704e4a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7700130af0 0x7f7700099700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7704e4a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7700130af0 0x7f7700099700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40402/0 (socket says 172.21.15.110:40402) 2024-09-15T04:02:55.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f770564b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76ec003a50 0x7f77001305b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52270/0 (socket says 172.21.15.110:52270) 2024-09-15T04:02:55.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7704e4a640 1 -- 172.21.15.110:0/3043865765 learned_addr learned my addr 172.21.15.110:0/3043865765 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:55.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7704e4a640 10 monclient(hunting): get_auth_request con 0x7f7700130af0 auth_method 0 2024-09-15T04:02:55.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7704e4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7704e4a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7704e4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:55.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f770564b640 10 monclient(hunting): get_auth_request con 0x7f76ec003a50 auth_method 0 2024-09-15T04:02:55.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f770564b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f770564b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.230 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f770564b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7704e4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7704e4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7705e4c640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7700099c40 0x7f770009a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7704e4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7705e4c640 10 monclient(hunting): get_auth_request con 0x7f7700099c40 auth_method 0 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7705e4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7705e4c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.211+0000 7f7705e4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.212+0000 7f7705e4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.212+0000 7f7705e4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.212+0000 7f7705e4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:55.231 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.212+0000 7f7704e4a640 10 monclient(hunting): handle_auth_done global_id 54723 payload 824 2024-09-15T04:02:55.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.212+0000 7f7704e4a640 1 -- 172.21.15.110:0/3043865765 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76ec003a50 msgr2=0x7f77001305b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.212+0000 7f7704e4a640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76ec003a50 0x7f77001305b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.212+0000 7f7704e4a640 1 -- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7700099c40 msgr2=0x7f770009a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.212+0000 7f7704e4a640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7700099c40 0x7f770009a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.212+0000 7f7704e4a640 10 monclient: _finish_hunting 0 2024-09-15T04:02:55.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.212+0000 7f7704e4a640 1 monclient: found mon.noname-a 2024-09-15T04:02:55.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.212+0000 7f7704e4a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:55.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.212+0000 7f7704e4a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:02:55.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.212+0000 7f7704e4a640 1 -- 172.21.15.110:0/3043865765 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770011c750 con 0x7f7700130af0 2024-09-15T04:02:55.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.212+0000 7f770564b640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f76ec003a50 0x7f77001305b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:02:55.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f7704e4a640 1 -- 172.21.15.110:0/3043865765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7700130af0 msgr2=0x7f7700099700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:55.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f7704e4a640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7700130af0 0x7f7700099700 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f76f00073a0 tx=0x7f76f0007af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:55.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f76f57fa640 1 -- 172.21.15.110:0/3043865765 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f0018030 con 0x7f7700130af0 2024-09-15T04:02:55.232 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f76f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f76f57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:02:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f76f57fa640 10 monclient: dump: 2024-09-15T04:02:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:55.233 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:55.233 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f76f57fa640 10 monclient: _finish_auth 0 2024-09-15T04:02:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f76f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f76f57fa640 1 -- 172.21.15.110:0/3043865765 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f000fbd0 con 0x7f7700130af0 2024-09-15T04:02:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f77078af640 5 monclient: authenticate success, global_id 54723 2024-09-15T04:02:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f77078af640 10 monclient: _renew_subs 2024-09-15T04:02:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f77078af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f77078af640 1 -- 172.21.15.110:0/3043865765 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7700137280 con 0x7f7700130af0 2024-09-15T04:02:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f76f57fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f76f57fa640 1 -- 172.21.15.110:0/3043865765 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f00133f0 con 0x7f7700130af0 2024-09-15T04:02:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f76f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f76f57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f76f57fa640 10 monclient: dump: 2024-09-15T04:02:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:55.234 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.235 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.235 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.235 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:55.235 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.235 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:55.235 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:55.235 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:55.235 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f76f77fe640 4 set_mon_vals no callback set 2024-09-15T04:02:55.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f77078af640 10 monclient: _renew_subs 2024-09-15T04:02:55.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f77078af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:55.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.214+0000 7f77078af640 1 -- 172.21.15.110:0/3043865765 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f770009b9a0 con 0x7f7700130af0 2024-09-15T04:02:55.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f76f57fa640 1 -- 172.21.15.110:0/3043865765 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f76f001b070 con 0x7f7700130af0 2024-09-15T04:02:55.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f77078af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:55.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f77078af640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f770009a5f0 0x7f7700095a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f77078af640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f77078af640 1 -- 172.21.15.110:0/3043865765 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7700063e80 con 0x7f770009a5f0 2024-09-15T04:02:55.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f76f57fa640 1 --2- 172.21.15.110:0/3043865765 >> v2:172.21.15.177:6812/28201 conn(0x7f76e4038780 0x7f76e403ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f76f57fa640 1 -- 172.21.15.110:0/3043865765 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f76f0019070 con 0x7f7700130af0 2024-09-15T04:02:55.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f770564b640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f770009a5f0 0x7f7700095a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f7705e4c640 1 --2- 172.21.15.110:0/3043865765 >> v2:172.21.15.177:6812/28201 conn(0x7f76e4038780 0x7f76e403ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f7705e4c640 10 monclient: get_auth_request con 0x7f76e4038780 auth_method 0 2024-09-15T04:02:55.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f770564b640 10 monclient: get_auth_request con 0x7f770009a5f0 auth_method 0 2024-09-15T04:02:55.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f770564b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f770564b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f770564b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:55.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:55.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:55.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.215+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:55.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.216+0000 7f770564b640 10 monclient(hunting): handle_auth_done global_id 5290 payload 824 2024-09-15T04:02:55.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.216+0000 7f770564b640 1 -- 172.21.15.110:0/3043865765 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f770011c8e0 con 0x7f770009a5f0 2024-09-15T04:02:55.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.218+0000 7f7705e4c640 1 --2- 172.21.15.110:0/3043865765 >> v2:172.21.15.177:6812/28201 conn(0x7f76e4038780 0x7f76e403ac30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f77000a4d70 tx=0x7f76f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:55.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.218+0000 7f770564b640 1 -- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f770009a5f0 msgr2=0x7f7700095a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:55.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.218+0000 7f770564b640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f770009a5f0 0x7f7700095a70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f76fc006e70 tx=0x7f76fc035000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:55.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.218+0000 7f76f57fa640 1 -- 172.21.15.110:0/3043865765 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc040070 con 0x7f770009a5f0 2024-09-15T04:02:55.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.218+0000 7f76f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:55.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.218+0000 7f76f57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:02:55.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.218+0000 7f76f57fa640 10 monclient: dump: 2024-09-15T04:02:55.238 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:55.238 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.238 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.238 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.238 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:55.238 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.238 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:55.238 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:55.238 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:55.238 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.218+0000 7f76f57fa640 1 -- 172.21.15.110:0/3043865765 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f76fc03b830 con 0x7f770009a5f0 2024-09-15T04:02:55.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.218+0000 7f76f57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:55.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.218+0000 7f76f57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:02:55.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.218+0000 7f76f57fa640 1 -- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f770009a5f0 msgr2=0x7f7700095a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.218+0000 7f76f57fa640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f770009a5f0 0x7f7700095a70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f76fc006e70 tx=0x7f76fc035000 comp rx=0 tx=0).stop 2024-09-15T04:02:55.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.218+0000 7f76f57fa640 1 -- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f770009a5f0 msgr2=0x7f7700095a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:02:55.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.218+0000 7f76f57fa640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f770009a5f0 0x7f7700095a70 secure :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f76fc006e70 tx=0x7f76fc035000 comp rx=0 tx=0).stop 2024-09-15T04:02:55.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.224+0000 7f77078af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:02:55.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.224+0000 7f77078af640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7700095fb0 0x7f77000963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.224+0000 7f77078af640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.224+0000 7f77078af640 1 -- 172.21.15.110:0/3043865765 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77000647e0 con 0x7f7700095fb0 2024-09-15T04:02:55.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.225+0000 7f770564b640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7700095fb0 0x7f77000963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.225+0000 7f770564b640 10 monclient: get_auth_request con 0x7f7700095fb0 auth_method 0 2024-09-15T04:02:55.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.225+0000 7f770564b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.225+0000 7f770564b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.225+0000 7f770564b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:55.241 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.225+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.225+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.225+0000 7f770564b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.225+0000 7f770564b640 10 monclient(hunting): handle_auth_done global_id 5293 payload 824 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.225+0000 7f770564b640 1 -- 172.21.15.110:0/3043865765 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f77000008d0 con 0x7f7700095fb0 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.227+0000 7f770564b640 1 -- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7700095fb0 msgr2=0x7f77000963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.227+0000 7f770564b640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7700095fb0 0x7f77000963b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009b70 tx=0x7f76fc040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.227+0000 7f76f57fa640 1 -- 172.21.15.110:0/3043865765 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76fc006250 con 0x7f7700095fb0 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.227+0000 7f76f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.227+0000 7f76f57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.227+0000 7f76f57fa640 10 monclient: dump: 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.242 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:55.243 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.243 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:55.243 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:55.243 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:55.243 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f76f57fa640 1 -- 172.21.15.110:0/3043865765 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f76fc0063b0 con 0x7f7700095fb0 2024-09-15T04:02:55.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f76f57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:02:55.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f76f57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:02:55.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f76f57fa640 1 -- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7700095fb0 msgr2=0x7f77000963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f76f57fa640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7700095fb0 0x7f77000963b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009b70 tx=0x7f76fc040000 comp rx=0 tx=0).stop 2024-09-15T04:02:55.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f76f57fa640 1 -- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7700095fb0 msgr2=0x7f77000963b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:02:55.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f76f57fa640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7700095fb0 0x7f77000963b0 secure :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009b70 tx=0x7f76fc040000 comp rx=0 tx=0).stop 2024-09-15T04:02:55.244 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:02:55.244 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:02:55.244 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:02:55.244 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T04:02:55.244 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 28, 2024-09-15T04:02:55.244 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:02:55.244 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:02:55.244 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:02:55.244 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:02:55.244 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:55.244 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 4, 2024-09-15T04:02:55.244 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:02:55.245 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:02:55.245 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:02:55.245 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:55.245 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:55.245 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:55.245 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:55.245 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:55.245 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:55.245 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:55.245 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:55.245 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:55.245 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:55.245 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:02:55.245 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:02:55.246 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:55.246 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:55.246 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:55.246 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:55.246 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:55.246 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:55.246 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:55.246 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:55.246 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:55.246 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:55.246 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.246 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:02:55.246 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:02:55.246 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:02:55.247 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:02:55.247 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:02:55.247 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:02:55.247 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:02:55.247 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:02:55.247 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:02:55.247 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:02:55.247 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:02:55.247 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:02:55.247 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:02:55.247 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:02:55.247 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:02:55.247 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:02:55.247 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:02:55.248 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:55.248 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:55.248 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:55.248 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:55.248 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:55.248 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:55.248 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:55.248 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:55.248 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:55.248 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:55.248 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:02:55.248 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.249 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:02:55.249 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.249 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:02:55.249 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:02:55.249 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:55.249 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:55.249 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.249 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:55.249 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:02:55.249 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:55.249 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.249 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.249 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:55.249 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:02:55.250 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:55.250 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:55.250 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:55.250 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.250 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:02:55.250 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:02:55.250 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:55.250 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:55.250 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:55.250 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.250 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.250 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:02:55.250 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:02:55.250 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:55.251 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:55.251 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.251 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:55.251 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:02:55.251 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:55.251 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.251 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.251 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:55.251 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:02:55.251 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:55.251 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:55.251 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:55.251 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.251 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:02:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:02:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:02:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:02:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:02:55.252 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:55.253 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.253 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.253 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:55.253 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:02:55.253 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:55.253 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:55.253 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:55.253 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.253 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:02:55.253 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:02:55.253 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:55.253 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:55.253 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:55.253 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:55.254 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 5 2024-09-15T04:02:55.255 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:55.255 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:55.255 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:02:55.255 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.255 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:55.255 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:55.255 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 6 2024-09-15T04:02:55.255 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:55.255 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:55.255 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": [ 2024-09-15T04:02:55.256 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:55.256 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:55.256 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:55.256 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:02:55.256 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:55.256 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:55.256 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:55.256 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:02:55.256 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:02:55.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f77078af640 1 -- 172.21.15.110:0/3043865765 >> v2:172.21.15.177:6812/28201 conn(0x7f76e4038780 msgr2=0x7f76e403ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f77078af640 1 --2- 172.21.15.110:0/3043865765 >> v2:172.21.15.177:6812/28201 conn(0x7f76e4038780 0x7f76e403ac30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f77000a4d70 tx=0x7f76f8009040 comp rx=0 tx=0).stop 2024-09-15T04:02:55.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f77078af640 10 monclient: shutdown 2024-09-15T04:02:55.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f77078af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:55.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f77078af640 1 -- 172.21.15.110:0/3043865765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7700130af0 msgr2=0x7f7700099700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f77078af640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7700130af0 0x7f7700099700 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f76f00073a0 tx=0x7f76f0007af0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f7705e4c640 1 -- 172.21.15.110:0/3043865765 reap_dead start 2024-09-15T04:02:55.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f77078af640 1 -- 172.21.15.110:0/3043865765 shutdown_connections 2024-09-15T04:02:55.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f77078af640 1 --2- 172.21.15.110:0/3043865765 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7700130af0 0x7f7700099700 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f77078af640 1 -- 172.21.15.110:0/3043865765 >> 172.21.15.110:0/3043865765 conn(0x7f770006e860 msgr2=0x7f770009d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:55.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f77078af640 1 -- 172.21.15.110:0/3043865765 shutdown_connections 2024-09-15T04:02:55.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.228+0000 7f77078af640 1 -- 172.21.15.110:0/3043865765 wait complete. 2024-09-15T04:02:55.258 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:02:55.259 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T04:02:55.259 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T04:02:55.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.352+0000 7f8c1d44f640 10 monclient: get_monmap_and_config 2024-09-15T04:02:55.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.352+0000 7f8c1d44f640 10 monclient: build_initial_monmap 2024-09-15T04:02:55.353 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.352+0000 7f8c1d44f640 10 monclient: monmap: 2024-09-15T04:02:55.353 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:55.353 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.353 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:02:55.353413+0000 2024-09-15T04:02:55.353 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:02:55.353413+0000 2024-09-15T04:02:55.353 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:55.353 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.354 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:02:55.354 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:02:55.354 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:02:55.354 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:02:55.354 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:02:55.354 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.352+0000 7f8c1d44f640 1 Processor -- start 2024-09-15T04:02:55.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 1 -- start start 2024-09-15T04:02:55.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 10 monclient: init 2024-09-15T04:02:55.356 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:55.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-15T04:02:55.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8c180a1bf0 addr v1:172.21.15.110:6789/0 2024-09-15T04:02:55.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f8c1809e780 0x7f8c1809eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 10 monclient(hunting): picked mon.noname-c con 0x7f8c1809e780 addr v2:172.21.15.177:3300/0 2024-09-15T04:02:55.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8c18136a40 0x7f8c18138e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 10 monclient(hunting): picked mon.noname-a con 0x7f8c18136a40 addr v2:172.21.15.110:3300/0 2024-09-15T04:02:55.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1809c9b0 con 0x7f8c180a1bf0 2024-09-15T04:02:55.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c180a0c20 con 0x7f8c18136a40 2024-09-15T04:02:55.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c1809cc90 con 0x7f8c1809e780 2024-09-15T04:02:55.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:55.357 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c1d44f640 10 monclient(hunting): authenticate will time out at 998.997549s 2024-09-15T04:02:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8c18136a40 0x7f8c18138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8c18136a40 0x7f8c18138e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40414/0 (socket says 172.21.15.110:40414) 2024-09-15T04:02:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c16ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8c180a1bf0 0x7f8c18131810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53784/0 (socket says 172.21.15.110:53784) 2024-09-15T04:02:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 1 -- 172.21.15.110:0/2798171132 learned_addr learned my addr 172.21.15.110:0/2798171132 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 10 monclient(hunting): get_auth_request con 0x7f8c18136a40 auth_method 0 2024-09-15T04:02:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c167fc640 1 --2- 172.21.15.110:0/2798171132 >> v2:172.21.15.177:3300/0 conn(0x7f8c1809e780 0x7f8c1809eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 10 monclient(hunting): handle_auth_done global_id 54726 payload 274 2024-09-15T04:02:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 1 -- 172.21.15.110:0/2798171132 >> v2:172.21.15.177:3300/0 conn(0x7f8c1809e780 msgr2=0x7f8c1809eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 1 --2- 172.21.15.110:0/2798171132 >> v2:172.21.15.177:3300/0 conn(0x7f8c1809e780 0x7f8c1809eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 1 -- 172.21.15.110:0/2798171132 >> v1:172.21.15.110:6789/0 conn(0x7f8c180a1bf0 legacy=0x7f8c18131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 10 monclient: _finish_hunting 0 2024-09-15T04:02:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 1 monclient: found mon.noname-a 2024-09-15T04:02:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:02:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c177fe640 1 -- 172.21.15.110:0/2798171132 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c18131f20 con 0x7f8c18136a40 2024-09-15T04:02:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c167fc640 10 monclient: get_auth_request con 0x7f8c1809e780 auth_method 0 2024-09-15T04:02:55.359 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.353+0000 7f8c167fc640 1 --2- 172.21.15.110:0/2798171132 >> v2:172.21.15.177:3300/0 conn(0x7f8c1809e780 0x7f8c1809eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:02:55.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.356+0000 7f8c177fe640 1 -- 172.21.15.110:0/2798171132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c18136a40 msgr2=0x7f8c18138e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:55.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.356+0000 7f8c177fe640 1 --2- 172.21.15.110:0/2798171132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c18136a40 0x7f8c18138e20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8c180a2bb0 tx=0x7f8c10009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9736b202b69cf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:55.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.356+0000 7f8c15ffb640 1 -- 172.21.15.110:0/2798171132 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c10007610 con 0x7f8c18136a40 2024-09-15T04:02:55.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.356+0000 7f8c15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:55.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c15ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:02:55.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c15ffb640 10 monclient: dump: 2024-09-15T04:02:55.365 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:55.365 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.365 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.366 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.366 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:55.366 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.366 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:55.366 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:55.366 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:55.366 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c15ffb640 10 monclient: _finish_auth 0 2024-09-15T04:02:55.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:55.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c15ffb640 1 -- 172.21.15.110:0/2798171132 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1002d050 con 0x7f8c18136a40 2024-09-15T04:02:55.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c1d44f640 5 monclient: authenticate success, global_id 54726 2024-09-15T04:02:55.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c1d44f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:02:55.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c15ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:55.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c1d44f640 10 monclient: get_monmap_and_config success 2024-09-15T04:02:55.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c15ffb640 1 -- 172.21.15.110:0/2798171132 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c100079e0 con 0x7f8c18136a40 2024-09-15T04:02:55.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:55.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c1d44f640 4 set_mon_vals no callback set 2024-09-15T04:02:55.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:55.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c15ffb640 10 monclient: dump: 2024-09-15T04:02:55.367 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:55.367 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.367 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.367 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.368 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:55.368 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.368 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:55.368 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:55.368 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:55.368 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c1d44f640 10 monclient: shutdown 2024-09-15T04:02:55.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c1d44f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:55.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c1d44f640 1 -- 172.21.15.110:0/2798171132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c18136a40 msgr2=0x7f8c18138e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c1d44f640 1 --2- 172.21.15.110:0/2798171132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c18136a40 0x7f8c18138e20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8c180a2bb0 tx=0x7f8c10009840 comp rx=0 tx=0).stop 2024-09-15T04:02:55.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c1d44f640 1 -- 172.21.15.110:0/2798171132 shutdown_connections 2024-09-15T04:02:55.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c1d44f640 1 --2- 172.21.15.110:0/2798171132 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c18136a40 0x7f8c18138e20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c1d44f640 1 --2- 172.21.15.110:0/2798171132 >> v2:172.21.15.177:3300/0 conn(0x7f8c1809e780 0x7f8c1809eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c1d44f640 1 -- 172.21.15.110:0/2798171132 >> 172.21.15.110:0/2798171132 conn(0x7f8c18062dd0 msgr2=0x7f8c180631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:55.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c1d44f640 1 -- 172.21.15.110:0/2798171132 shutdown_connections 2024-09-15T04:02:55.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.357+0000 7f8c1d44f640 1 -- 172.21.15.110:0/2798171132 wait complete. 2024-09-15T04:02:55.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.358+0000 7f8c1d44f640 10 monclient: build_initial_monmap 2024-09-15T04:02:55.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.358+0000 7f8c1d44f640 10 monclient: monmap: 2024-09-15T04:02:55.369 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:55.369 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:02:55.369 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:02:55.370 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:02:55.370 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:55.370 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.370 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:02:55.370 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:02:55.370 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:02:55.370 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.358+0000 7f8c1d44f640 1 Processor -- start 2024-09-15T04:02:55.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.358+0000 7f8c1d44f640 1 -- start start 2024-09-15T04:02:55.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.358+0000 7f8c1d44f640 10 monclient: init 2024-09-15T04:02:55.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.358+0000 7f8c1d44f640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:55.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.358+0000 7f8c1d44f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T04:02:55.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c1d44f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c1809e780 0x7f8c18130630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c1d44f640 10 monclient(hunting): picked mon.noname-b con 0x7f8c1809e780 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:02:55.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c1d44f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c180a1bf0 0x7f8c18099950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c1d44f640 10 monclient(hunting): picked mon.noname-a con 0x7f8c180a1bf0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:55.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c1d44f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c18136a40 0x7f8c18099e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c1d44f640 10 monclient(hunting): picked mon.noname-c con 0x7f8c18136a40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:55.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c1d44f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c1d44f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1811cd00 con 0x7f8c180a1bf0 2024-09-15T04:02:55.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c1d44f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c1d44f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180a2f30 con 0x7f8c1809e780 2024-09-15T04:02:55.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c1d44f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:55.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c1d44f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1811ca20 con 0x7f8c18136a40 2024-09-15T04:02:55.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c1d44f640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:55.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c1d44f640 10 monclient(hunting): authenticate will time out at 999.003068s 2024-09-15T04:02:55.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c180a1bf0 0x7f8c18099950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c180a1bf0 0x7f8c18099950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40416/0 (socket says 172.21.15.110:40416) 2024-09-15T04:02:55.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 1 -- 172.21.15.110:0/3447825535 learned_addr learned my addr 172.21.15.110:0/3447825535 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:55.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 10 monclient(hunting): get_auth_request con 0x7f8c180a1bf0 auth_method 0 2024-09-15T04:02:55.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:55.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c177fe640 1 --2- 172.21.15.110:0/3447825535 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c18136a40 0x7f8c18099e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c177fe640 10 monclient(hunting): get_auth_request con 0x7f8c18136a40 auth_method 0 2024-09-15T04:02:55.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:55.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:55.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:55.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:55.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c16ffd640 1 --2- 172.21.15.110:0/3447825535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c1809e780 0x7f8c18130630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:55.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:55.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:55.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c16ffd640 10 monclient(hunting): get_auth_request con 0x7f8c1809e780 auth_method 0 2024-09-15T04:02:55.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:55.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:55.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:55.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 10 monclient(hunting): handle_auth_done global_id 54729 payload 824 2024-09-15T04:02:55.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 1 -- 172.21.15.110:0/3447825535 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c18136a40 msgr2=0x7f8c18099e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 1 --2- 172.21.15.110:0/3447825535 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c18136a40 0x7f8c18099e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 1 -- 172.21.15.110:0/3447825535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c1809e780 msgr2=0x7f8c18130630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 1 --2- 172.21.15.110:0/3447825535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c1809e780 0x7f8c18130630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 10 monclient: _finish_hunting 0 2024-09-15T04:02:55.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 1 monclient: found mon.noname-a 2024-09-15T04:02:55.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:55.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:02:55.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c167fc640 1 -- 172.21.15.110:0/3447825535 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c10009ae0 con 0x7f8c180a1bf0 2024-09-15T04:02:55.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.359+0000 7f8c16ffd640 1 --2- 172.21.15.110:0/3447825535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c1809e780 0x7f8c18130630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T04:02:55.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8c167fc640 1 -- 172.21.15.110:0/3447825535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c180a1bf0 msgr2=0x7f8c18099950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:55.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8c167fc640 1 --2- 172.21.15.110:0/3447825535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c180a1bf0 0x7f8c18099950 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c00c880 tx=0x7f8c0c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:55.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8bfeffd640 1 -- 172.21.15.110:0/3447825535 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c00ce10 con 0x7f8c180a1bf0 2024-09-15T04:02:55.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8bfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:55.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8bfeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:02:55.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8bfeffd640 10 monclient: dump: 2024-09-15T04:02:55.374 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:55.374 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8bfeffd640 10 monclient: _finish_auth 0 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8bfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8bfeffd640 1 -- 172.21.15.110:0/3447825535 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0c007a40 con 0x7f8c180a1bf0 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8c1d44f640 5 monclient: authenticate success, global_id 54729 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8c1d44f640 10 monclient: _renew_subs 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8c1d44f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:55.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8c1d44f640 1 -- 172.21.15.110:0/3447825535 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c1811c740 con 0x7f8c180a1bf0 2024-09-15T04:02:55.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8bfeffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:55.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8bfeffd640 1 -- 172.21.15.110:0/3447825535 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c007e00 con 0x7f8c180a1bf0 2024-09-15T04:02:55.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8bfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:55.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8bfeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:55.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8bfeffd640 10 monclient: dump: 2024-09-15T04:02:55.376 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:55.376 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:55.376 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.376 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:55.376 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:55.376 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:55.376 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:55.376 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:55.376 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:55.377 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:55.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8c1d44f640 10 monclient: _renew_subs 2024-09-15T04:02:55.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8c1d44f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:55.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8c1d44f640 1 -- 172.21.15.110:0/3447825535 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c1809b560 con 0x7f8c180a1bf0 2024-09-15T04:02:55.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.361+0000 7f8c157fa640 4 set_mon_vals no callback set 2024-09-15T04:02:55.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.362+0000 7f8bfeffd640 1 -- 172.21.15.110:0/3447825535 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f8c0c011600 con 0x7f8c180a1bf0 2024-09-15T04:02:55.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.362+0000 7f8c1d44f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:55.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.362+0000 7f8c1d44f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:55.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.362+0000 7f8c1d44f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:55.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.362+0000 7f8c1d44f640 1 -- 172.21.15.110:0/3447825535 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1811ce40 con 0x7f8c180a1bf0 2024-09-15T04:02:55.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.362+0000 7f8bfeffd640 1 --2- 172.21.15.110:0/3447825535 >> v2:172.21.15.177:6812/28201 conn(0x7f8c04038730 0x7f8c0403abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:55.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.362+0000 7f8bfeffd640 1 -- 172.21.15.110:0/3447825535 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8c0c04b650 con 0x7f8c180a1bf0 2024-09-15T04:02:55.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.363+0000 7f8c16ffd640 1 --2- 172.21.15.110:0/3447825535 >> v2:172.21.15.177:6812/28201 conn(0x7f8c04038730 0x7f8c0403abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:55.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.363+0000 7f8c16ffd640 10 monclient: get_auth_request con 0x7f8c04038730 auth_method 0 2024-09-15T04:02:55.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.365+0000 7f8c16ffd640 1 --2- 172.21.15.110:0/3447825535 >> v2:172.21.15.177:6812/28201 conn(0x7f8c04038730 0x7f8c0403abe0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8c08005130 tx=0x7f8c08009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:55.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.367+0000 7f8bfeffd640 1 -- 172.21.15.110:0/3447825535 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f8c1811ce40 con 0x7f8c180a1bf0 2024-09-15T04:02:55.378 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.367+0000 7f8bfeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:55.379 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.367+0000 7f8bfeffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:02:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.589+0000 7f8c1d44f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T04:02:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.589+0000 7f8c1d44f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T04:02:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.589+0000 7f8c1d44f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.589+0000 7f8c1d44f640 1 -- 172.21.15.110:0/3447825535 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f8c1811ce40 con 0x7f8c180a1bf0 2024-09-15T04:02:55.590 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.590+0000 7f8bfeffd640 1 -- 172.21.15.110:0/3447825535 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f8c1811ce40 con 0x7f8c180a1bf0 2024-09-15T04:02:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.590+0000 7f8bfeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T04:02:55.591 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.590+0000 7f8bfeffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:02:55.591 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.590+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.591 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.590+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=515454349,config=3140514009,health=660307592,kv=1242700903,logm=2333333137}) 2024-09-15T04:02:55.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.590+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=11,kv=6,logm=68} crc {auth=741665640,config=3140514009,health=2809466272,kv=120571890,logm=2373922347}) 2024-09-15T04:02:55.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.590+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.590+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=8,logm=68} crc {auth=515454349,config=3140514009,health=660307592,kv=1242700903,logm=2333333137}) 2024-09-15T04:02:55.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.590+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=12,kv=8,logm=65} crc {auth=741665640,config=3140514009,health=25822461,kv=1242700903,logm=716395719}) 2024-09-15T04:02:55.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513469596}) 2024-09-15T04:02:55.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2506779102}) 2024-09-15T04:02:55.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3513469596}) 2024-09-15T04:02:55.592 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2178478587}) 2024-09-15T04:02:55.593 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.593 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=959338316}) 2024-09-15T04:02:55.593 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2133652890}) 2024-09-15T04:02:55.593 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.593 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=959338316}) 2024-09-15T04:02:55.593 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.591+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3503549546}) 2024-09-15T04:02:55.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f8c1d44f640 1 -- 172.21.15.110:0/3447825535 >> v2:172.21.15.177:6812/28201 conn(0x7f8c04038730 msgr2=0x7f8c0403abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f8c1d44f640 1 --2- 172.21.15.110:0/3447825535 >> v2:172.21.15.177:6812/28201 conn(0x7f8c04038730 0x7f8c0403abe0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8c08005130 tx=0x7f8c08009040 comp rx=0 tx=0).stop 2024-09-15T04:02:55.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f8c1d44f640 10 monclient: shutdown 2024-09-15T04:02:55.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f8c1d44f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:55.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f8c1d44f640 1 -- 172.21.15.110:0/3447825535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c180a1bf0 msgr2=0x7f8c18099950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:55.594 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f8c1d44f640 1 --2- 172.21.15.110:0/3447825535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c180a1bf0 0x7f8c18099950 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c00c880 tx=0x7f8c0c008040 comp rx=0 tx=0).stop 2024-09-15T04:02:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f8c1d44f640 1 -- 172.21.15.110:0/3447825535 shutdown_connections 2024-09-15T04:02:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f8c1d44f640 1 --2- 172.21.15.110:0/3447825535 >> v2:172.21.15.177:6812/28201 conn(0x7f8c04038730 0x7f8c0403abe0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f8c1d44f640 1 --2- 172.21.15.110:0/3447825535 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8c18136a40 0x7f8c18099e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f8c1d44f640 1 --2- 172.21.15.110:0/3447825535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8c180a1bf0 0x7f8c18099950 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f8c1d44f640 1 --2- 172.21.15.110:0/3447825535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8c1809e780 0x7f8c18130630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:55.595 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f8c1d44f640 1 -- 172.21.15.110:0/3447825535 >> 172.21.15.110:0/3447825535 conn(0x7f8c18062dd0 msgr2=0x7f8c180a4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:55.595 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.596 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2697588320}) 2024-09-15T04:02:55.596 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3140617078}) 2024-09-15T04:02:55.596 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.596 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2697588320}) 2024-09-15T04:02:55.596 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1488426643}) 2024-09-15T04:02:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f8c1d44f640 1 -- 172.21.15.110:0/3447825535 shutdown_connections 2024-09-15T04:02:55.596 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:55.592+0000 7f8c1d44f640 1 -- 172.21.15.110:0/3447825535 wait complete. 2024-09-15T04:02:55.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.593+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.593+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414265727}) 2024-09-15T04:02:55.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.593+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1452936281}) 2024-09-15T04:02:55.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.593+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.593+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414265727}) 2024-09-15T04:02:55.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.593+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=47962667}) 2024-09-15T04:02:55.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3072061901}) 2024-09-15T04:02:55.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1294280273}) 2024-09-15T04:02:55.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.600 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3072061901}) 2024-09-15T04:02:55.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=435642852}) 2024-09-15T04:02:55.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193570981}) 2024-09-15T04:02:55.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4254105103}) 2024-09-15T04:02:55.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193570981}) 2024-09-15T04:02:55.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.594+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1228926638}) 2024-09-15T04:02:55.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.595+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.595+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434839759}) 2024-09-15T04:02:55.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.595+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2640469441}) 2024-09-15T04:02:55.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.595+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.595+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434839759}) 2024-09-15T04:02:55.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.595+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3308042469}) 2024-09-15T04:02:55.601 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=424292124}) 2024-09-15T04:02:55.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=328052583}) 2024-09-15T04:02:55.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=424292124}) 2024-09-15T04:02:55.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.596+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2506568659}) 2024-09-15T04:02:55.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.597+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.597+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=59,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1} crc {logm=776549990,mdsmap=1531151339,mgr=3394955505,mgr_command_descs=3545737492,mgr_metadata=3491211682,mgrstat=3995344722,mon_config_key=3532925585,monmap=1680381720,osd_metadata=3281624269,osd_pg_creating=2365294520,osd_snap=3823785925}) 2024-09-15T04:02:55.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.597+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=58,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=1392789129,mdsmap=1531151339,mgr=4138502149,mgr_command_descs=3545737492,mgr_metadata=3491211682,mgrstat=4177297909,mon_config_key=3532925585,monmap=3538901833,osd_metadata=3281624269,osd_pg_creating=2365294520,osd_snap=3178633393}) 2024-09-15T04:02:55.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.597+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.597+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=59,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=1} crc {logm=776549990,mdsmap=1531151339,mgr=3394955505,mgr_command_descs=3545737492,mgr_metadata=3491211682,mgrstat=3995344722,mon_config_key=3532925585,monmap=1680381720,osd_metadata=3281624269,osd_pg_creating=2365294520,osd_snap=3823785925}) 2024-09-15T04:02:55.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.597+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=58,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=1162358785,mdsmap=1531151339,mgr=4138502149,mgr_command_descs=3545737492,mgr_metadata=3491211682,mgrstat=1673615732,mon_config_key=3532925585,monmap=1680381720,osd_metadata=3281624269,osd_pg_creating=2365294520,osd_snap=3178633393}) 2024-09-15T04:02:55.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.598+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.602 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.598+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=1,osdmap=99} crc {osd_snap=956710772,osdmap=2642164656}) 2024-09-15T04:02:55.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.598+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=1,osdmap=99} crc {osd_snap=956710772,osdmap=1611788503}) 2024-09-15T04:02:55.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.598+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.598+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=1,osdmap=99} crc {osd_snap=956710772,osdmap=2642164656}) 2024-09-15T04:02:55.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.598+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=1,osdmap=99} crc {osd_snap=956710772,osdmap=240788078}) 2024-09-15T04:02:55.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.599+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.599+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2316384210}) 2024-09-15T04:02:55.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.599+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3960999652}) 2024-09-15T04:02:55.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.599+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.599+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2316384210}) 2024-09-15T04:02:55.603 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.599+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4204140832}) 2024-09-15T04:02:55.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.600+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.600+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2947020377}) 2024-09-15T04:02:55.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.600+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3861717356}) 2024-09-15T04:02:55.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.600+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.600+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2947020377}) 2024-09-15T04:02:55.604 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.600+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=437213551}) 2024-09-15T04:02:55.605 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T04:02:55.608 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.607+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.608 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.607+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=40} crc {osdmap=301257143}) 2024-09-15T04:02:55.609 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.608+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=42} crc {osdmap=2281313300}) 2024-09-15T04:02:55.609 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.608+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:02:55.609 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.608+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=40} crc {osdmap=301257143}) 2024-09-15T04:02:55.609 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:02:55.608+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=38} crc {osdmap=2460596904}) 2024-09-15T04:02:56.606 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T04:02:56.606 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:02:56.673 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:56.673 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32592]: starting. 2024-09-15T04:02:56.673 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32592]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32557 2024-09-15T04:02:56.673 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32592]: created object 0... 2024-09-15T04:02:56.673 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32592]: created object 25... 2024-09-15T04:02:56.673 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32592]: created object 49... 2024-09-15T04:02:56.673 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32592]: finishing. 2024-09-15T04:02:56.673 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32592]: shutting down. 2024-09-15T04:02:56.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.706+0000 7f818ae4f640 10 monclient: get_monmap_and_config 2024-09-15T04:02:56.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.706+0000 7f818ae4f640 10 monclient: build_initial_monmap 2024-09-15T04:02:56.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.706+0000 7f818ae4f640 10 monclient: monmap: 2024-09-15T04:02:56.707 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:56.707 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:56.707 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:02:56.707195+0000 2024-09-15T04:02:56.707 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:02:56.707195+0000 2024-09-15T04:02:56.707 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:56.707 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:56.707 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:02:56.707 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:02:56.708 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:02:56.708 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:02:56.708 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:02:56.708 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:56.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.706+0000 7f818ae4f640 1 Processor -- start 2024-09-15T04:02:56.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f818ae4f640 1 -- start start 2024-09-15T04:02:56.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f818ae4f640 10 monclient: init 2024-09-15T04:02:56.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f818ae4f640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:56.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f818ae4f640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-15T04:02:56.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f818ae4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81840a1a60 addr v1:172.21.15.110:6789/0 2024-09-15T04:02:56.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f818ae4f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 0x7f8184131a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:56.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f818ae4f640 10 monclient(hunting): picked mon.noname-b con 0x7f8184131640 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:56.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f818ae4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8184131f50 addr v1:172.21.15.177:6789/0 2024-09-15T04:02:56.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f818ae4f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81840a1260 con 0x7f81840a1a60 2024-09-15T04:02:56.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f818ae4f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f818409c870 con 0x7f8184131f50 2024-09-15T04:02:56.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f818ae4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:56.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f818ae4f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81840a0c60 con 0x7f8184131640 2024-09-15T04:02:56.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f818ae4f640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:56.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f818ae4f640 10 monclient(hunting): authenticate will time out at 1000.351618s 2024-09-15T04:02:56.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8188beb640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f81840a1a60 0x7f818409d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53790/0 (socket says 172.21.15.110:53790) 2024-09-15T04:02:56.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8188beb640 1 -- 172.21.15.110:0/817744724 learned_addr learned my addr 172.21.15.110:0/817744724 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:02:56.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 1 --2- 172.21.15.110:0/817744724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 0x7f8184131a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:56.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 10 monclient(hunting): get_auth_request con 0x7f8184131640 auth_method 0 2024-09-15T04:02:56.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:56.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:56.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:56.711 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:56.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:56.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:56.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 10 monclient(hunting): handle_auth_done global_id 64112 payload 274 2024-09-15T04:02:56.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 1 -- 172.21.15.110:0/817744724 >> v1:172.21.15.177:6789/0 conn(0x7f8184131f50 legacy=0x7f8184138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:56.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 1 -- 172.21.15.110:0/817744724 >> v1:172.21.15.110:6789/0 conn(0x7f81840a1a60 legacy=0x7f818409d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:56.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 10 monclient: _finish_hunting 0 2024-09-15T04:02:56.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 1 monclient: found mon.noname-b 2024-09-15T04:02:56.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:56.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:02:56.712 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.707+0000 7f8183fff640 1 -- 172.21.15.110:0/817744724 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818409cb10 con 0x7f8184131640 2024-09-15T04:02:56.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.710+0000 7f8183fff640 1 -- 172.21.15.110:0/817744724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 msgr2=0x7f8184131a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:56.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.710+0000 7f8183fff640 1 --2- 172.21.15.110:0/817744724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 0x7f8184131a10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8174009e90 tx=0x7f8174009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a23b12f11fc78043 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:56.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.710+0000 7f81837fe640 1 -- 172.21.15.110:0/817744724 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8174007840 con 0x7f8184131640 2024-09-15T04:02:56.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.710+0000 7f81837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:56.718 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.710+0000 7f81837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:02:56.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.710+0000 7f81837fe640 10 monclient: dump: 2024-09-15T04:02:56.719 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:56.719 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:56.719 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:56.719 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:56.719 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:56.719 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:56.719 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:56.719 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:56.719 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:56.719 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:56.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.710+0000 7f81837fe640 10 monclient: _finish_auth 0 2024-09-15T04:02:56.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.710+0000 7f81837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:56.719 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.710+0000 7f818ae4f640 5 monclient: authenticate success, global_id 64112 2024-09-15T04:02:56.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.710+0000 7f818ae4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:02:56.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.710+0000 7f81837fe640 1 -- 172.21.15.110:0/817744724 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817402d050 con 0x7f8184131640 2024-09-15T04:02:56.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f81837fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:56.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f818ae4f640 10 monclient: get_monmap_and_config success 2024-09-15T04:02:56.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f818ae4f640 4 set_mon_vals no callback set 2024-09-15T04:02:56.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f81837fe640 1 -- 172.21.15.110:0/817744724 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8174007bd0 con 0x7f8184131640 2024-09-15T04:02:56.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f81837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:56.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f81837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:02:56.720 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f81837fe640 10 monclient: dump: 2024-09-15T04:02:56.720 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:56.720 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:56.720 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:56.720 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:56.720 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:56.721 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:56.721 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:56.721 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:56.721 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:56.721 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:56.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f818ae4f640 10 monclient: shutdown 2024-09-15T04:02:56.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f818ae4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:56.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f818ae4f640 1 -- 172.21.15.110:0/817744724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 msgr2=0x7f8184131a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:56.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f818ae4f640 1 --2- 172.21.15.110:0/817744724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 0x7f8184131a10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8174009e90 tx=0x7f8174009780 comp rx=0 tx=0).stop 2024-09-15T04:02:56.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f818ae4f640 1 -- 172.21.15.110:0/817744724 shutdown_connections 2024-09-15T04:02:56.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f818ae4f640 1 --2- 172.21.15.110:0/817744724 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 0x7f8184131a10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:56.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f818ae4f640 1 -- 172.21.15.110:0/817744724 >> 172.21.15.110:0/817744724 conn(0x7f8184063270 msgr2=0x7f8184063670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:56.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f818ae4f640 1 -- 172.21.15.110:0/817744724 shutdown_connections 2024-09-15T04:02:56.721 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.711+0000 7f818ae4f640 1 -- 172.21.15.110:0/817744724 wait complete. 2024-09-15T04:02:56.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.712+0000 7f818ae4f640 10 monclient: build_initial_monmap 2024-09-15T04:02:56.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.712+0000 7f818ae4f640 10 monclient: monmap: 2024-09-15T04:02:56.722 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:56.722 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:02:56.722 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:02:56.722 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:02:56.722 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:56.722 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:56.722 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:02:56.722 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:02:56.722 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:02:56.722 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:56.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.712+0000 7f818ae4f640 1 Processor -- start 2024-09-15T04:02:56.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.712+0000 7f818ae4f640 1 -- start start 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.712+0000 7f818ae4f640 10 monclient: init 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a1a60 0x7f8184130c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 10 monclient(hunting): picked mon.noname-a con 0x7f81840a1a60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 0x7f8184095a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 10 monclient(hunting): picked mon.noname-c con 0x7f8184131640 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8184131f50 0x7f8184095fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 10 monclient(hunting): picked mon.noname-b con 0x7f8184131f50 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81840a1450 con 0x7f81840a1a60 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8184099430 con 0x7f8184131f50 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:56.723 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8184096b10 con 0x7f8184131640 2024-09-15T04:02:56.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:56.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f818ae4f640 10 monclient(hunting): authenticate will time out at 1000.357245s 2024-09-15T04:02:56.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a1a60 0x7f8184130c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:56.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8183fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 0x7f8184095a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:56.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a1a60 0x7f8184130c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40442/0 (socket says 172.21.15.110:40442) 2024-09-15T04:02:56.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 1 -- 172.21.15.110:0/2265843224 learned_addr learned my addr 172.21.15.110:0/2265843224 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:56.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 10 monclient(hunting): get_auth_request con 0x7f81840a1a60 auth_method 0 2024-09-15T04:02:56.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:56.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:56.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:56.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8183fff640 10 monclient(hunting): get_auth_request con 0x7f8184131640 auth_method 0 2024-09-15T04:02:56.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8183fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:56.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8183fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:56.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8183fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:56.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f81893ec640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8184131f50 0x7f8184095fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:56.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:56.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:56.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:56.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f81893ec640 10 monclient(hunting): get_auth_request con 0x7f8184131f50 auth_method 0 2024-09-15T04:02:56.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f81893ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:56.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f81893ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:56.725 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f81893ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:56.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8183fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:56.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8183fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:56.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8183fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:56.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f81893ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:56.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f81893ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:56.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f81893ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:56.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 10 monclient(hunting): handle_auth_done global_id 54732 payload 824 2024-09-15T04:02:56.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 1 -- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 msgr2=0x7f8184095a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:56.726 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 0x7f8184095a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:56.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 1 -- 172.21.15.110:0/2265843224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8184131f50 msgr2=0x7f8184095fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:56.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8184131f50 0x7f8184095fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:56.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 10 monclient: _finish_hunting 0 2024-09-15T04:02:56.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 1 monclient: found mon.noname-a 2024-09-15T04:02:56.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:56.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:02:56.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 1 -- 172.21.15.110:0/2265843224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8174009a20 con 0x7f81840a1a60 2024-09-15T04:02:56.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8183fff640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 0x7f8184095a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:02:56.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 1 -- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a1a60 msgr2=0x7f8184130c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T04:02:56.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 1 -- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a1a60 msgr2=0x7f8184130c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T04:02:56.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a1a60 0x7f8184130c50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f817c00c5e0 tx=0x7f817c004960 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:02:56.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.713+0000 7f8188beb640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a1a60 0x7f8184130c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:02:56.728 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:56.728 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32594]: starting. 2024-09-15T04:02:56.728 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32594]: listing objects. 2024-09-15T04:02:56.728 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32594]: listed object 0... 2024-09-15T04:02:56.728 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32594]: listed object 25... 2024-09-15T04:02:56.728 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32594]: saw 50 objects 2024-09-15T04:02:56.728 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32594]: shutting down. 2024-09-15T04:02:56.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a1a60 0x7f8184130c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:56.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 10 monclient: get_auth_request con 0x7f81840a1a60 auth_method 0 2024-09-15T04:02:56.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 0 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a1a60 0x7f8184130c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T04:02:56.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a1a60 0x7f8184130c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:56.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:56.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:56.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 1 -- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a1a60 msgr2=0x7f8184130c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:02:56.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81840a1a60 0x7f8184130c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:56.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T04:02:56.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8164003460 0x7f81640058a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:56.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f8164003460 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:56.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8184131f50 0x7f8164008460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:56.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f8184131f50 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:02:56.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 0x7f816400ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:56.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f8184131640 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:56.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:56.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 1 -- 172.21.15.110:0/2265843224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816400b7e0 con 0x7f8164003460 2024-09-15T04:02:56.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:56.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 1 -- 172.21.15.110:0/2265843224 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816400b950 con 0x7f8184131f50 2024-09-15T04:02:56.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:56.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 1 -- 172.21.15.110:0/2265843224 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816400bb10 con 0x7f8184131640 2024-09-15T04:02:56.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8180ff9640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8164003460 0x7f81640058a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 10 monclient(hunting): get_auth_request con 0x7f8164003460 auth_method 0 2024-09-15T04:02:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:02:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f81893ec640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 0x7f816400ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f81893ec640 10 monclient(hunting): get_auth_request con 0x7f8184131640 auth_method 0 2024-09-15T04:02:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f81893ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f81893ec640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f81893ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:02:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:56.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:02:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f81893ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f81893ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f81893ec640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:02:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 10 monclient(hunting): handle_auth_done global_id 54732 payload 293 2024-09-15T04:02:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 1 -- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 msgr2=0x7f816400ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 0x7f816400ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 1 -- 172.21.15.110:0/2265843224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8184131f50 msgr2=0x7f8164008460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8184131f50 0x7f8164008460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 10 monclient: _finish_hunting 0 2024-09-15T04:02:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 1 monclient: found mon.noname-a 2024-09-15T04:02:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:02:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f8188beb640 1 -- 172.21.15.110:0/2265843224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816400bd90 con 0x7f8164003460 2024-09-15T04:02:56.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.914+0000 7f81893ec640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 0x7f816400ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:02:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8188beb640 1 -- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8164003460 msgr2=0x7f81640058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8188beb640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8164003460 0x7f81640058a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f81840a1590 tx=0x7f817c004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8180ff9640 1 -- 172.21.15.110:0/2265843224 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c007300 con 0x7f8164003460 2024-09-15T04:02:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8180ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8180ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:02:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8180ff9640 10 monclient: dump: 2024-09-15T04:02:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:56.922 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8180ff9640 10 monclient: _finish_auth 0 2024-09-15T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8180ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8180ff9640 1 -- 172.21.15.110:0/2265843224 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f817c007460 con 0x7f8164003460 2024-09-15T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f818ae4f640 5 monclient: authenticate success, global_id 54732 2024-09-15T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f818ae4f640 10 monclient: _renew_subs 2024-09-15T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f818ae4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f818ae4f640 1 -- 172.21.15.110:0/2265843224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81840973f0 con 0x7f8164003460 2024-09-15T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8180ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8180ff9640 1 -- 172.21.15.110:0/2265843224 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c012420 con 0x7f8164003460 2024-09-15T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8180ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:56.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8180ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:56.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8180ff9640 10 monclient: dump: 2024-09-15T04:02:56.924 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:56.924 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:56.924 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:56.924 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:56.924 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:56.924 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:56.924 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:56.924 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:56.924 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:56.924 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:56.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f818ae4f640 10 monclient: _renew_subs 2024-09-15T04:02:56.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f818ae4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:56.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f8182ffd640 4 set_mon_vals no callback set 2024-09-15T04:02:56.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.917+0000 7f818ae4f640 1 -- 172.21.15.110:0/2265843224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f818409a890 con 0x7f8164003460 2024-09-15T04:02:56.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.918+0000 7f8180ff9640 1 -- 172.21.15.110:0/2265843224 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f817c012790 con 0x7f8164003460 2024-09-15T04:02:56.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.918+0000 7f818ae4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:56.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.918+0000 7f818ae4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:56.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.918+0000 7f818ae4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:56.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.918+0000 7f818ae4f640 1 -- 172.21.15.110:0/2265843224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8184098890 con 0x7f8164003460 2024-09-15T04:02:56.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.918+0000 7f8180ff9640 1 --2- 172.21.15.110:0/2265843224 >> v2:172.21.15.177:6812/28201 conn(0x7f81840a1a60 0x7f8164042b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:56.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.918+0000 7f8180ff9640 1 -- 172.21.15.110:0/2265843224 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f817c04be20 con 0x7f8164003460 2024-09-15T04:02:56.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.918+0000 7f8183fff640 1 --2- 172.21.15.110:0/2265843224 >> v2:172.21.15.177:6812/28201 conn(0x7f81840a1a60 0x7f8164042b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:56.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.918+0000 7f8183fff640 10 monclient: get_auth_request con 0x7f81840a1a60 auth_method 0 2024-09-15T04:02:56.926 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.921+0000 7f8183fff640 1 --2- 172.21.15.110:0/2265843224 >> v2:172.21.15.177:6812/28201 conn(0x7f81840a1a60 0x7f8164042b50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f81840a0a10 tx=0x7f8174037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:56.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.923+0000 7f8180ff9640 1 -- 172.21.15.110:0/2265843224 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f817c025580 con 0x7f8164003460 2024-09-15T04:02:56.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.923+0000 7f8180ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:56.927 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:56.923+0000 7f8180ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:02:57.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.135+0000 7f818ae4f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:02:57.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.135+0000 7f818ae4f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:02:57.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.135+0000 7f818ae4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:02:57.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.135+0000 7f818ae4f640 1 -- 172.21.15.110:0/2265843224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f818413a200 con 0x7f8164003460 2024-09-15T04:02:57.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.136+0000 7f8180ff9640 1 -- 172.21.15.110:0/2265843224 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f817c026ba0 con 0x7f8164003460 2024-09-15T04:02:57.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.136+0000 7f8180ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:02:57.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.136+0000 7f8180ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:02:57.138 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:02:57.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.138+0000 7f818ae4f640 1 -- 172.21.15.110:0/2265843224 >> v2:172.21.15.177:6812/28201 conn(0x7f81840a1a60 msgr2=0x7f8164042b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.138+0000 7f818ae4f640 1 --2- 172.21.15.110:0/2265843224 >> v2:172.21.15.177:6812/28201 conn(0x7f81840a1a60 0x7f8164042b50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f81840a0a10 tx=0x7f8174037000 comp rx=0 tx=0).stop 2024-09-15T04:02:57.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.138+0000 7f818ae4f640 10 monclient: shutdown 2024-09-15T04:02:57.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.138+0000 7f818ae4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:57.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.138+0000 7f818ae4f640 1 -- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8164003460 msgr2=0x7f81640058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.138+0000 7f818ae4f640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8164003460 0x7f81640058a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f81840a1590 tx=0x7f817c004960 comp rx=0 tx=0).stop 2024-09-15T04:02:57.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.138+0000 7f818ae4f640 1 -- 172.21.15.110:0/2265843224 shutdown_connections 2024-09-15T04:02:57.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.138+0000 7f818ae4f640 1 --2- 172.21.15.110:0/2265843224 >> v2:172.21.15.177:6812/28201 conn(0x7f81840a1a60 0x7f8164042b50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.138+0000 7f818ae4f640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8184131640 0x7f816400ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.138+0000 7f818ae4f640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8184131f50 0x7f8164008460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.138+0000 7f818ae4f640 1 --2- 172.21.15.110:0/2265843224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8164003460 0x7f81640058a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.138+0000 7f818ae4f640 1 -- 172.21.15.110:0/2265843224 >> 172.21.15.110:0/2265843224 conn(0x7f8184063270 msgr2=0x7f818409dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:57.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.138+0000 7f818ae4f640 1 -- 172.21.15.110:0/2265843224 shutdown_connections 2024-09-15T04:02:57.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.138+0000 7f818ae4f640 1 -- 172.21.15.110:0/2265843224 wait complete. 2024-09-15T04:02:57.149 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T04:02:57.149 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T04:02:57.149 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T04:02:57.149 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T04:02:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.242+0000 7fe3abe34640 10 monclient: get_monmap_and_config 2024-09-15T04:02:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.242+0000 7fe3abe34640 10 monclient: build_initial_monmap 2024-09-15T04:02:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.242+0000 7fe3abe34640 10 monclient: monmap: 2024-09-15T04:02:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:02:57.243160+0000 2024-09-15T04:02:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:02:57.243160+0000 2024-09-15T04:02:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.245 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:02:57.246 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:02:57.246 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:02:57.246 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:02:57.246 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:02:57.246 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.242+0000 7fe3abe34640 1 Processor -- start 2024-09-15T04:02:57.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.242+0000 7fe3abe34640 1 -- start start 2024-09-15T04:02:57.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.242+0000 7fe3abe34640 10 monclient: init 2024-09-15T04:02:57.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3abe34640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:57.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3abe34640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-15T04:02:57.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3abe34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3a40a1a10 addr v1:172.21.15.110:6789/0 2024-09-15T04:02:57.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3abe34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3a409ee80 addr v1:172.21.15.177:6789/0 2024-09-15T04:02:57.246 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3abe34640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3a4131b90 0x7fe3a413a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3abe34640 10 monclient(hunting): picked mon.noname-b con 0x7fe3a4131b90 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:57.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3abe34640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a409f2c0 con 0x7fe3a40a1a10 2024-09-15T04:02:57.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3abe34640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a40a0850 con 0x7fe3a409ee80 2024-09-15T04:02:57.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3abe34640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3abe34640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a409c9a0 con 0x7fe3a4131b90 2024-09-15T04:02:57.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3abe34640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:57.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3abe34640 10 monclient(hunting): authenticate will time out at 1000.887430s 2024-09-15T04:02:57.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3a4131b90 0x7fe3a413a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3a9bd0640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe3a40a1a10 0x7fe3a409e770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:53796/0 (socket says 172.21.15.110:53796) 2024-09-15T04:02:57.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3a9bd0640 1 -- 172.21.15.110:0/1849714651 learned_addr learned my addr 172.21.15.110:0/1849714651 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:02:57.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 10 monclient(hunting): get_auth_request con 0x7fe3a4131b90 auth_method 0 2024-09-15T04:02:57.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 10 monclient(hunting): handle_auth_done global_id 64118 payload 274 2024-09-15T04:02:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 1 -- 172.21.15.110:0/1849714651 >> v1:172.21.15.177:6789/0 conn(0x7fe3a409ee80 legacy=0x7fe3a41369e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 1 -- 172.21.15.110:0/1849714651 >> v1:172.21.15.110:6789/0 conn(0x7fe3a40a1a10 legacy=0x7fe3a409e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 10 monclient: _finish_hunting 0 2024-09-15T04:02:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 1 monclient: found mon.noname-b 2024-09-15T04:02:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:02:57.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.243+0000 7fe3aa3d1640 1 -- 172.21.15.110:0/1849714651 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a409cc80 con 0x7fe3a4131b90 2024-09-15T04:02:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3aa3d1640 1 -- 172.21.15.110:0/1849714651 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3a4131b90 msgr2=0x7fe3a413a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3aa3d1640 1 --2- 172.21.15.110:0/1849714651 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3a4131b90 0x7fe3a413a110 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009880 tx=0x7fe3a00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e178ab71ecb1499 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3a8bce640 1 -- 172.21.15.110:0/1849714651 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0032440 con 0x7fe3a4131b90 2024-09-15T04:02:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3a8bce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3a8bce640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:02:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3a8bce640 10 monclient: dump: 2024-09-15T04:02:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.255 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.256 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.256 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.256 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3a8bce640 10 monclient: _finish_auth 0 2024-09-15T04:02:57.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3a8bce640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:57.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3a8bce640 1 -- 172.21.15.110:0/1849714651 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a00325a0 con 0x7fe3a4131b90 2024-09-15T04:02:57.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3a8bce640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:57.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3a8bce640 1 -- 172.21.15.110:0/1849714651 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0032940 con 0x7fe3a4131b90 2024-09-15T04:02:57.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3abe34640 5 monclient: authenticate success, global_id 64118 2024-09-15T04:02:57.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3abe34640 10 monclient: get_monmap_and_config success 2024-09-15T04:02:57.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3abe34640 4 set_mon_vals no callback set 2024-09-15T04:02:57.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3a8bce640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3a8bce640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:02:57.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.246+0000 7fe3a8bce640 10 monclient: dump: 2024-09-15T04:02:57.257 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.257 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.257 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.257 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.257 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.257 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.257 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.257 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.257 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.257 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.247+0000 7fe3abe34640 10 monclient: shutdown 2024-09-15T04:02:57.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.247+0000 7fe3abe34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:57.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.247+0000 7fe3abe34640 1 -- 172.21.15.110:0/1849714651 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3a4131b90 msgr2=0x7fe3a413a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.247+0000 7fe3abe34640 1 --2- 172.21.15.110:0/1849714651 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3a4131b90 0x7fe3a413a110 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009880 tx=0x7fe3a00090f0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.247+0000 7fe3abe34640 1 -- 172.21.15.110:0/1849714651 shutdown_connections 2024-09-15T04:02:57.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.247+0000 7fe3abe34640 1 --2- 172.21.15.110:0/1849714651 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3a4131b90 0x7fe3a413a110 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.247+0000 7fe3abe34640 1 -- 172.21.15.110:0/1849714651 >> 172.21.15.110:0/1849714651 conn(0x7fe3a406e860 msgr2=0x7fe3a406ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.247+0000 7fe3abe34640 1 -- 172.21.15.110:0/1849714651 shutdown_connections 2024-09-15T04:02:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.247+0000 7fe3abe34640 1 -- 172.21.15.110:0/1849714651 wait complete. 2024-09-15T04:02:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.247+0000 7fe3abe34640 10 monclient: build_initial_monmap 2024-09-15T04:02:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.247+0000 7fe3abe34640 10 monclient: monmap: 2024-09-15T04:02:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:02:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:02:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:02:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.259 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:02:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:02:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:02:57.260 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 1 Processor -- start 2024-09-15T04:02:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 1 -- start start 2024-09-15T04:02:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 10 monclient: init 2024-09-15T04:02:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T04:02:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3a409ee80 0x7fe3a4130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 10 monclient(hunting): picked mon.noname-b con 0x7fe3a409ee80 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:02:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a40a1a10 0x7fe3a4131020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 10 monclient(hunting): picked mon.noname-a con 0x7fe3a40a1a10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:57.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3a4131b90 0x7fe3a4140e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 10 monclient(hunting): picked mon.noname-c con 0x7fe3a4131b90 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:57.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a411f4b0 con 0x7fe3a40a1a10 2024-09-15T04:02:57.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a411eef0 con 0x7fe3a409ee80 2024-09-15T04:02:57.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a411f1d0 con 0x7fe3a4131b90 2024-09-15T04:02:57.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:57.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3abe34640 10 monclient(hunting): authenticate will time out at 1000.892801s 2024-09-15T04:02:57.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3aa3d1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3a4131b90 0x7fe3a4140e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3a93cf640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a40a1a10 0x7fe3a4131020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3aa3d1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3a4131b90 0x7fe3a4140e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52326/0 (socket says 172.21.15.110:52326) 2024-09-15T04:02:57.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3aa3d1640 1 -- 172.21.15.110:0/1178002715 learned_addr learned my addr 172.21.15.110:0/1178002715 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3a93cf640 10 monclient(hunting): get_auth_request con 0x7fe3a40a1a10 auth_method 0 2024-09-15T04:02:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3a93cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3a93cf640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3a93cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3aa3d1640 10 monclient(hunting): get_auth_request con 0x7fe3a4131b90 auth_method 0 2024-09-15T04:02:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3aa3d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3aa3d1640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3aa3d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3a93cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.248+0000 7fe3a93cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3a93cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3aa3d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3aa3d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3aa3d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3a9bd0640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3a409ee80 0x7fe3a4130ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3a9bd0640 10 monclient(hunting): get_auth_request con 0x7fe3a409ee80 auth_method 0 2024-09-15T04:02:57.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3a9bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3a9bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3a9bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3aa3d1640 10 monclient(hunting): handle_auth_done global_id 64121 payload 824 2024-09-15T04:02:57.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3aa3d1640 1 -- 172.21.15.110:0/1178002715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3a409ee80 msgr2=0x7fe3a4130ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3aa3d1640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe3a409ee80 0x7fe3a4130ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3aa3d1640 1 -- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a40a1a10 msgr2=0x7fe3a4131020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3aa3d1640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a40a1a10 0x7fe3a4131020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3aa3d1640 10 monclient: _finish_hunting 0 2024-09-15T04:02:57.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3aa3d1640 1 monclient: found mon.noname-c 2024-09-15T04:02:57.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3aa3d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3aa3d1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:02:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3aa3d1640 1 -- 172.21.15.110:0/1178002715 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a0009390 con 0x7fe3a4131b90 2024-09-15T04:02:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.249+0000 7fe3a93cf640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a40a1a10 0x7fe3a4131020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:02:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe3aa3d1640 1 -- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3a4131b90 msgr2=0x7fe3a4140e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe3aa3d1640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3a4131b90 0x7fe3a4140e50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe3a00097d0 tx=0x7fe3a0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe399ffb640 1 -- 172.21.15.110:0/1178002715 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0031c70 con 0x7fe3a4131b90 2024-09-15T04:02:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe399ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe399ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:02:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe399ffb640 10 monclient: dump: 2024-09-15T04:02:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.264 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe399ffb640 10 monclient: _finish_auth 0 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe399ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe399ffb640 1 -- 172.21.15.110:0/1178002715 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a0031dd0 con 0x7fe3a4131b90 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe3abe34640 5 monclient: authenticate success, global_id 64121 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe3abe34640 10 monclient: _renew_subs 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe3abe34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe3abe34640 1 -- 172.21.15.110:0/1178002715 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a411dcb0 con 0x7fe3a4131b90 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe399ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:57.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe399ffb640 1 -- 172.21.15.110:0/1178002715 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a003b610 con 0x7fe3a4131b90 2024-09-15T04:02:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe399ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe399ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:02:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe399ffb640 10 monclient: dump: 2024-09-15T04:02:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.266 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.267 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe39bfff640 4 set_mon_vals no callback set 2024-09-15T04:02:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe3abe34640 10 monclient: _renew_subs 2024-09-15T04:02:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe3abe34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:02:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.251+0000 7fe3abe34640 1 -- 172.21.15.110:0/1178002715 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a4141390 con 0x7fe3a4131b90 2024-09-15T04:02:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.252+0000 7fe3abe34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.252+0000 7fe3abe34640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411b730 0x7fe3a411bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.252+0000 7fe3abe34640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.252+0000 7fe3abe34640 1 -- 172.21.15.110:0/1178002715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a411edb0 con 0x7fe3a411b730 2024-09-15T04:02:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.252+0000 7fe399ffb640 1 -- 172.21.15.110:0/1178002715 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fe3a0039040 con 0x7fe3a4131b90 2024-09-15T04:02:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.252+0000 7fe3a9bd0640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411b730 0x7fe3a411bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.252+0000 7fe3a9bd0640 10 monclient: get_auth_request con 0x7fe3a411b730 auth_method 0 2024-09-15T04:02:57.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.252+0000 7fe3a9bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.252+0000 7fe3a9bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.252+0000 7fe3a9bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.252+0000 7fe3a9bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.252+0000 7fe3a9bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.253+0000 7fe3a9bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.253+0000 7fe399ffb640 1 --2- 172.21.15.110:0/1178002715 >> v2:172.21.15.177:6812/28201 conn(0x7fe388038700 0x7fe38803abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.253+0000 7fe399ffb640 1 -- 172.21.15.110:0/1178002715 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fe3a006bed0 con 0x7fe3a4131b90 2024-09-15T04:02:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.253+0000 7fe3a9bd0640 10 monclient(hunting): handle_auth_done global_id 54738 payload 824 2024-09-15T04:02:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.253+0000 7fe3a9bd0640 1 -- 172.21.15.110:0/1178002715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3a40647e0 con 0x7fe3a411b730 2024-09-15T04:02:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.253+0000 7fe3a93cf640 1 --2- 172.21.15.110:0/1178002715 >> v2:172.21.15.177:6812/28201 conn(0x7fe388038700 0x7fe38803abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.253+0000 7fe3a93cf640 10 monclient: get_auth_request con 0x7fe388038700 auth_method 0 2024-09-15T04:02:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.255+0000 7fe3a9bd0640 1 -- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411b730 msgr2=0x7fe3a411bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.255+0000 7fe3a9bd0640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411b730 0x7fe3a411bb00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00ec30 tx=0x7fe39c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.255+0000 7fe399ffb640 1 -- 172.21.15.110:0/1178002715 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c00ee50 con 0x7fe3a411b730 2024-09-15T04:02:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.255+0000 7fe399ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.255+0000 7fe399ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.255+0000 7fe399ffb640 10 monclient: dump: 2024-09-15T04:02:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.271 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.272 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.255+0000 7fe399ffb640 1 -- 172.21.15.110:0/1178002715 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe39c00cc60 con 0x7fe3a411b730 2024-09-15T04:02:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.255+0000 7fe399ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.255+0000 7fe399ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:02:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.255+0000 7fe399ffb640 1 -- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411b730 msgr2=0x7fe3a411bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.255+0000 7fe399ffb640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411b730 0x7fe3a411bb00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00ec30 tx=0x7fe39c009040 comp rx=0 tx=0).stop 2024-09-15T04:02:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.255+0000 7fe399ffb640 1 -- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411b730 msgr2=0x7fe3a411bb00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:02:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.255+0000 7fe399ffb640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411b730 0x7fe3a411bb00 secure :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00ec30 tx=0x7fe39c009040 comp rx=0 tx=0).stop 2024-09-15T04:02:57.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.256+0000 7fe3a93cf640 1 --2- 172.21.15.110:0/1178002715 >> v2:172.21.15.177:6812/28201 conn(0x7fe388038700 0x7fe38803abb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4141ea0 tx=0x7fe394009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.261+0000 7fe3abe34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:02:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.261+0000 7fe3abe34640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411c040 0x7fe3a4097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.261+0000 7fe3abe34640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.261+0000 7fe3abe34640 1 -- 172.21.15.110:0/1178002715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a4063e80 con 0x7fe3a411c040 2024-09-15T04:02:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.261+0000 7fe3a9bd0640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411c040 0x7fe3a4097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.261+0000 7fe3a9bd0640 10 monclient: get_auth_request con 0x7fe3a411c040 auth_method 0 2024-09-15T04:02:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.261+0000 7fe3a9bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.261+0000 7fe3a9bd0640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.261+0000 7fe3a9bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.261+0000 7fe3a9bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.261+0000 7fe3a9bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.261+0000 7fe3a9bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.261+0000 7fe3a9bd0640 10 monclient(hunting): handle_auth_done global_id 54741 payload 824 2024-09-15T04:02:57.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.261+0000 7fe3a9bd0640 1 -- 172.21.15.110:0/1178002715 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe3a40a1570 con 0x7fe3a411c040 2024-09-15T04:02:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.263+0000 7fe3a9bd0640 1 -- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411c040 msgr2=0x7fe3a4097a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.263+0000 7fe3a9bd0640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411c040 0x7fe3a4097a80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00ec30 tx=0x7fe39c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe399ffb640 1 -- 172.21.15.110:0/1178002715 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c01d9b0 con 0x7fe3a411c040 2024-09-15T04:02:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe399ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe399ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe399ffb640 10 monclient: dump: 2024-09-15T04:02:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.274 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.275 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.275 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe399ffb640 1 -- 172.21.15.110:0/1178002715 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4579 (secure 0 0 0) 0x7fe39c01db10 con 0x7fe3a411c040 2024-09-15T04:02:57.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe399ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:02:57.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe399ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:02:57.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe399ffb640 1 -- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411c040 msgr2=0x7fe3a4097a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe399ffb640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411c040 0x7fe3a4097a80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00ec30 tx=0x7fe39c01b000 comp rx=0 tx=0).stop 2024-09-15T04:02:57.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe399ffb640 1 -- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411c040 msgr2=0x7fe3a4097a80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:02:57.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe399ffb640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe3a411c040 0x7fe3a4097a80 secure :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00ec30 tx=0x7fe39c01b000 comp rx=0 tx=0).stop 2024-09-15T04:02:57.276 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:02:57.276 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:02:57.276 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:02:57.276 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "leader", 2024-09-15T04:02:57.276 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 28, 2024-09-15T04:02:57.276 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:02:57.276 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:02:57.276 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:02:57.277 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:02:57.277 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.277 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 6, 2024-09-15T04:02:57.277 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:02:57.277 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:02:57.277 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:02:57.278 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:57.278 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:57.278 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:57.278 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:57.278 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:57.278 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:57.278 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:57.278 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:57.278 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:57.279 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.279 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:02:57.279 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:02:57.279 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:57.279 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:57.279 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:57.279 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:57.279 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:57.279 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:57.279 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:57.279 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:57.279 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:57.280 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.280 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.280 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:02:57.280 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:02:57.280 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:02:57.280 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:02:57.280 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:02:57.280 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:02:57.280 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:02:57.280 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:02:57.281 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:02:57.281 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:02:57.281 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:02:57.281 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:02:57.281 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:02:57.281 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:02:57.281 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:02:57.281 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:02:57.281 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:02:57.281 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:57.281 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:57.281 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:57.281 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:57.281 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:57.282 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:57.282 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:57.282 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:57.282 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:57.282 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.282 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:02:57.282 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.282 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:02:57.282 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.282 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:02:57.282 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:02:57.282 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:57.282 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:57.283 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.283 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:57.283 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:02:57.283 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.283 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.283 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.283 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:57.283 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:02:57.284 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.284 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.284 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.284 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.284 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:02:57.284 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:02:57.284 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:57.284 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:57.284 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:57.284 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.284 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.284 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:02:57.284 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:02:57.285 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:57.285 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:57.285 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.285 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:57.285 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:02:57.285 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.285 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.285 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.285 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:57.285 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:02:57.285 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.285 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.285 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.285 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.286 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:02:57.286 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:02:57.286 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:57.286 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:57.286 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:57.286 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.287 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.287 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:02:57.287 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:02:57.287 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:57.287 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:57.287 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.287 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:57.287 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:02:57.287 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.287 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.287 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.288 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:57.288 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:02:57.289 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.289 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.289 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.289 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.289 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:02:57.289 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:02:57.289 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:57.289 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:57.289 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:57.289 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.289 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.289 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.289 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:02:57.289 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:02:57.290 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.290 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:57.290 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:57.290 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:02:57.290 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.290 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.290 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T04:02:57.290 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.290 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:57.291 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:57.291 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:02:57.291 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.291 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.292 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:02:57.292 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.292 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:57.292 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:57.292 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 2 2024-09-15T04:02:57.292 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.292 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.292 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.292 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:02:57.292 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:02:57.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe3abe34640 1 -- 172.21.15.110:0/1178002715 >> v2:172.21.15.177:6812/28201 conn(0x7fe388038700 msgr2=0x7fe38803abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe3abe34640 1 --2- 172.21.15.110:0/1178002715 >> v2:172.21.15.177:6812/28201 conn(0x7fe388038700 0x7fe38803abb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4141ea0 tx=0x7fe394009060 comp rx=0 tx=0).stop 2024-09-15T04:02:57.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe3abe34640 10 monclient: shutdown 2024-09-15T04:02:57.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe3abe34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:57.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe3abe34640 1 -- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3a4131b90 msgr2=0x7fe3a4140e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe3abe34640 1 --2- 172.21.15.110:0/1178002715 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe3a4131b90 0x7fe3a4140e50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe3a00097d0 tx=0x7fe3a0032000 comp rx=0 tx=0).stop 2024-09-15T04:02:57.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe3aa3d1640 1 -- 172.21.15.110:0/1178002715 reap_dead start 2024-09-15T04:02:57.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe3aa3d1640 1 -- 172.21.15.110:0/1178002715 reap_dead start 2024-09-15T04:02:57.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe3abe34640 1 -- 172.21.15.110:0/1178002715 shutdown_connections 2024-09-15T04:02:57.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe3abe34640 1 -- 172.21.15.110:0/1178002715 >> 172.21.15.110:0/1178002715 conn(0x7fe3a406e860 msgr2=0x7fe3a409d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:57.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe3abe34640 1 -- 172.21.15.110:0/1178002715 shutdown_connections 2024-09-15T04:02:57.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.264+0000 7fe3abe34640 1 -- 172.21.15.110:0/1178002715 wait complete. 2024-09-15T04:02:57.295 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:02:57.295 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T04:02:57.295 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T04:02:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.387+0000 7fe86e8c9640 10 monclient: get_monmap_and_config 2024-09-15T04:02:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.387+0000 7fe86e8c9640 10 monclient: build_initial_monmap 2024-09-15T04:02:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.387+0000 7fe86e8c9640 10 monclient: monmap: 2024-09-15T04:02:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:57.388 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:02:57.388714+0000 2024-09-15T04:02:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:02:57.388714+0000 2024-09-15T04:02:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:02:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:02:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:02:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:02:57.389 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:02:57.389 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 1 Processor -- start 2024-09-15T04:02:57.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 1 -- start start 2024-09-15T04:02:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 10 monclient: init 2024-09-15T04:02:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-15T04:02:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fe86809b0d0 0x7fe86806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 10 monclient(hunting): picked mon.noname-c con 0x7fe86809b0d0 addr v2:172.21.15.177:3300/0 2024-09-15T04:02:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 0x7fe868067500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe868067130 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe868067a40 addr v1:172.21.15.177:6789/0 2024-09-15T04:02:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8680a1330 con 0x7fe868067a40 2024-09-15T04:02:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe86809a290 con 0x7fe86809b0d0 2024-09-15T04:02:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.392 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe868099fb0 con 0x7fe868067130 2024-09-15T04:02:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.388+0000 7fe86e8c9640 10 monclient(hunting): authenticate will time out at 1001.032969s 2024-09-15T04:02:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 0x7fe868067500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 0x7fe868067500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52336/0 (socket says 172.21.15.110:52336) 2024-09-15T04:02:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 1 -- 172.21.15.110:0/221231046 learned_addr learned my addr 172.21.15.110:0/221231046 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 10 monclient(hunting): get_auth_request con 0x7fe868067130 auth_method 0 2024-09-15T04:02:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.393 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe867fff640 1 --2- 172.21.15.110:0/221231046 >> v2:172.21.15.177:3300/0 conn(0x7fe86809b0d0 0x7fe86806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 10 monclient(hunting): handle_auth_done global_id 64124 payload 274 2024-09-15T04:02:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 1 -- 172.21.15.110:0/221231046 >> v2:172.21.15.177:3300/0 conn(0x7fe86809b0d0 msgr2=0x7fe86806e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 1 --2- 172.21.15.110:0/221231046 >> v2:172.21.15.177:3300/0 conn(0x7fe86809b0d0 0x7fe86806e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 1 -- 172.21.15.110:0/221231046 >> v1:172.21.15.177:6789/0 conn(0x7fe868067a40 legacy=0x7fe8681382b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 10 monclient: _finish_hunting 0 2024-09-15T04:02:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 1 monclient: found mon.noname-b 2024-09-15T04:02:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:02:57.394 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.389+0000 7fe8677fe640 1 -- 172.21.15.110:0/221231046 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86809a570 con 0x7fe868067130 2024-09-15T04:02:57.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe8677fe640 1 -- 172.21.15.110:0/221231046 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 msgr2=0x7fe868067500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:57.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe8677fe640 1 --2- 172.21.15.110:0/221231046 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 0x7fe868067500 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009910 tx=0x7fe85c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d91e02251de2a1c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe866ffd640 1 -- 172.21.15.110:0/221231046 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c032440 con 0x7fe868067130 2024-09-15T04:02:57.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe866ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe866ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:02:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe866ffd640 10 monclient: dump: 2024-09-15T04:02:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.401 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.401 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe866ffd640 10 monclient: _finish_auth 0 2024-09-15T04:02:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe866ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe86e8c9640 5 monclient: authenticate success, global_id 64124 2024-09-15T04:02:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe866ffd640 1 -- 172.21.15.110:0/221231046 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe85c0325a0 con 0x7fe868067130 2024-09-15T04:02:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe86e8c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:02:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe866ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe86e8c9640 10 monclient: get_monmap_and_config success 2024-09-15T04:02:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe86e8c9640 4 set_mon_vals no callback set 2024-09-15T04:02:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe866ffd640 1 -- 172.21.15.110:0/221231046 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c032940 con 0x7fe868067130 2024-09-15T04:02:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe866ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe866ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:02:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe866ffd640 10 monclient: dump: 2024-09-15T04:02:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.402 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe86e8c9640 10 monclient: shutdown 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe86e8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe86e8c9640 1 -- 172.21.15.110:0/221231046 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 msgr2=0x7fe868067500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.392+0000 7fe86e8c9640 1 --2- 172.21.15.110:0/221231046 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 0x7fe868067500 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009910 tx=0x7fe85c0091d0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.393+0000 7fe86e8c9640 1 -- 172.21.15.110:0/221231046 shutdown_connections 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.393+0000 7fe86e8c9640 1 --2- 172.21.15.110:0/221231046 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 0x7fe868067500 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.393+0000 7fe86e8c9640 1 --2- 172.21.15.110:0/221231046 >> v2:172.21.15.177:3300/0 conn(0x7fe86809b0d0 0x7fe86806e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.393+0000 7fe86e8c9640 1 -- 172.21.15.110:0/221231046 >> 172.21.15.110:0/221231046 conn(0x7fe86809e570 msgr2=0x7fe86809e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.393+0000 7fe86e8c9640 1 -- 172.21.15.110:0/221231046 shutdown_connections 2024-09-15T04:02:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.393+0000 7fe86e8c9640 1 -- 172.21.15.110:0/221231046 wait complete. 2024-09-15T04:02:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.393+0000 7fe86e8c9640 10 monclient: build_initial_monmap 2024-09-15T04:02:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.393+0000 7fe86e8c9640 10 monclient: monmap: 2024-09-15T04:02:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:02:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:02:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:02:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:02:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:02:57.404 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:02:57.405 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.393+0000 7fe86e8c9640 1 Processor -- start 2024-09-15T04:02:57.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 1 -- start start 2024-09-15T04:02:57.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 10 monclient: init 2024-09-15T04:02:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T04:02:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 0x7fe86813d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 10 monclient(hunting): picked mon.noname-c con 0x7fe868067130 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe868067a40 0x7fe86813cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 10 monclient(hunting): picked mon.noname-b con 0x7fe868067a40 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:02:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe86809b0d0 0x7fe868140e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 10 monclient(hunting): picked mon.noname-a con 0x7fe86809b0d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8681228b0 con 0x7fe86809b0d0 2024-09-15T04:02:57.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8681222f0 con 0x7fe868067a40 2024-09-15T04:02:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe86811fe60 con 0x7fe868067130 2024-09-15T04:02:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86e8c9640 10 monclient(hunting): authenticate will time out at 1001.038459s 2024-09-15T04:02:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 0x7fe86813d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 0x7fe86813d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:52346/0 (socket says 172.21.15.110:52346) 2024-09-15T04:02:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 1 -- 172.21.15.110:0/507495473 learned_addr learned my addr 172.21.15.110:0/507495473 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86ce66640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe86809b0d0 0x7fe868140e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 10 monclient(hunting): get_auth_request con 0x7fe868067130 auth_method 0 2024-09-15T04:02:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.409 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86ce66640 10 monclient(hunting): get_auth_request con 0x7fe86809b0d0 auth_method 0 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86ce66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86ce66640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86ce66640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86ce66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86ce66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86ce66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe8677fe640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe868067a40 0x7fe86813cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe8677fe640 10 monclient(hunting): get_auth_request con 0x7fe868067a40 auth_method 0 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe8677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe8677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.410 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 10 monclient(hunting): handle_auth_done global_id 64127 payload 824 2024-09-15T04:02:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 1 -- 172.21.15.110:0/507495473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe868067a40 msgr2=0x7fe86813cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe868067a40 0x7fe86813cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 1 -- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe86809b0d0 msgr2=0x7fe868140e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe86809b0d0 0x7fe868140e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 10 monclient: _finish_hunting 0 2024-09-15T04:02:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 1 monclient: found mon.noname-c 2024-09-15T04:02:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:02:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe867fff640 1 -- 172.21.15.110:0/507495473 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c009450 con 0x7fe868067130 2024-09-15T04:02:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.394+0000 7fe86ce66640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe86809b0d0 0x7fe868140e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:02:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe867fff640 1 -- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 msgr2=0x7fe86813d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe867fff640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 0x7fe86813d8c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe85800c8b0 tx=0x7fe858008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe84ffff640 1 -- 172.21.15.110:0/507495473 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85800ce40 con 0x7fe868067130 2024-09-15T04:02:57.411 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe84ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe84ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:02:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe84ffff640 10 monclient: dump: 2024-09-15T04:02:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.412 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe84ffff640 10 monclient: _finish_auth 0 2024-09-15T04:02:57.412 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe84ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe84ffff640 1 -- 172.21.15.110:0/507495473 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe858007a40 con 0x7fe868067130 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe86e8c9640 5 monclient: authenticate success, global_id 64127 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe86e8c9640 10 monclient: _renew_subs 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe86e8c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe86e8c9640 1 -- 172.21.15.110:0/507495473 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe868122010 con 0x7fe868067130 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe84ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe84ffff640 1 -- 172.21.15.110:0/507495473 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe858007de0 con 0x7fe868067130 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe84ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe84ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe84ffff640 10 monclient: dump: 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.413 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.414 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe86e8c9640 10 monclient: _renew_subs 2024-09-15T04:02:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe86e8c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:02:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe86e8c9640 1 -- 172.21.15.110:0/507495473 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe86813d650 con 0x7fe868067130 2024-09-15T04:02:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.397+0000 7fe8667fc640 4 set_mon_vals no callback set 2024-09-15T04:02:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe86e8c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe86e8c9640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868121140 0x7fe868141370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe86e8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe86e8c9640 1 -- 172.21.15.110:0/507495473 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe868068c10 con 0x7fe868121140 2024-09-15T04:02:57.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe84ffff640 1 -- 172.21.15.110:0/507495473 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fe858011600 con 0x7fe868067130 2024-09-15T04:02:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe8677fe640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868121140 0x7fe868141370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe8677fe640 10 monclient: get_auth_request con 0x7fe868121140 auth_method 0 2024-09-15T04:02:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe8677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe8677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe8677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe8677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe8677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe84ffff640 1 --2- 172.21.15.110:0/507495473 >> v2:172.21.15.177:6812/28201 conn(0x7fe854038700 0x7fe85403abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe8677fe640 10 monclient(hunting): handle_auth_done global_id 64130 payload 824 2024-09-15T04:02:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe84ffff640 1 -- 172.21.15.110:0/507495473 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 3894+0+0 (secure 0 0 0) 0x7fe85804b5e0 con 0x7fe868067130 2024-09-15T04:02:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe8677fe640 1 -- 172.21.15.110:0/507495473 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe868122740 con 0x7fe868121140 2024-09-15T04:02:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe86ce66640 1 --2- 172.21.15.110:0/507495473 >> v2:172.21.15.177:6812/28201 conn(0x7fe854038700 0x7fe85403abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.398+0000 7fe86ce66640 10 monclient: get_auth_request con 0x7fe854038700 auth_method 0 2024-09-15T04:02:57.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.400+0000 7fe8677fe640 1 -- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868121140 msgr2=0x7fe868141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:57.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.400+0000 7fe8677fe640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868121140 0x7fe868141370 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe85c034000 tx=0x7fe85c032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.400+0000 7fe84ffff640 1 -- 172.21.15.110:0/507495473 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c03c970 con 0x7fe868121140 2024-09-15T04:02:57.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.400+0000 7fe84ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.400+0000 7fe84ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:02:57.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.400+0000 7fe84ffff640 10 monclient: dump: 2024-09-15T04:02:57.418 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.418 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.418 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.418 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.418 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.419 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.419 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.419 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.419 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.419 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.400+0000 7fe84ffff640 1 -- 172.21.15.110:0/507495473 <== mon.2 v2:172.21.15.110:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7fe85c03cad0 con 0x7fe868121140 2024-09-15T04:02:57.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.400+0000 7fe84ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:57.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.400+0000 7fe84ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:02:57.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.400+0000 7fe84ffff640 1 -- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868121140 msgr2=0x7fe868141370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.400+0000 7fe84ffff640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868121140 0x7fe868141370 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe85c034000 tx=0x7fe85c032000 comp rx=0 tx=0).stop 2024-09-15T04:02:57.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.400+0000 7fe84ffff640 1 -- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868121140 msgr2=0x7fe868141370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:02:57.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.401+0000 7fe84ffff640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868121140 0x7fe868141370 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.401+0000 7fe86ce66640 1 --2- 172.21.15.110:0/507495473 >> v2:172.21.15.177:6812/28201 conn(0x7fe854038700 0x7fe85403abb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe86813de40 tx=0x7fe860009780 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.406+0000 7fe86e8c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:02:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.406+0000 7fe86e8c9640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe86813e240 0x7fe8681418b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.406+0000 7fe86e8c9640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.406+0000 7fe86e8c9640 1 -- 172.21.15.110:0/507495473 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe868122530 con 0x7fe86813e240 2024-09-15T04:02:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.406+0000 7fe8677fe640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe86813e240 0x7fe8681418b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.406+0000 7fe8677fe640 10 monclient: get_auth_request con 0x7fe86813e240 auth_method 0 2024-09-15T04:02:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.406+0000 7fe8677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.406+0000 7fe8677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.406+0000 7fe8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.406+0000 7fe8677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.406+0000 7fe8677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.406+0000 7fe8677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.406+0000 7fe8677fe640 10 monclient(hunting): handle_auth_done global_id 64133 payload 824 2024-09-15T04:02:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.406+0000 7fe8677fe640 1 -- 172.21.15.110:0/507495473 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe868122a20 con 0x7fe86813e240 2024-09-15T04:02:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe8677fe640 1 -- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe86813e240 msgr2=0x7fe8681418b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe8677fe640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe86813e240 0x7fe8681418b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009210 tx=0x7fe85c03c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe84ffff640 1 -- 172.21.15.110:0/507495473 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe85c03bcc0 con 0x7fe86813e240 2024-09-15T04:02:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe84ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe84ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:02:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe84ffff640 10 monclient: dump: 2024-09-15T04:02:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.421 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.422 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.422 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.422 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.422 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe84ffff640 1 -- 172.21.15.110:0/507495473 <== mon.2 v2:172.21.15.110:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7fe85c03be20 con 0x7fe86813e240 2024-09-15T04:02:57.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe84ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:02:57.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe84ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:02:57.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe84ffff640 1 -- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe86813e240 msgr2=0x7fe8681418b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe84ffff640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe86813e240 0x7fe8681418b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009210 tx=0x7fe85c03c000 comp rx=0 tx=0).stop 2024-09-15T04:02:57.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe84ffff640 1 -- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe86813e240 msgr2=0x7fe8681418b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:02:57.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe84ffff640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe86813e240 0x7fe8681418b0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009210 tx=0x7fe85c03c000 comp rx=0 tx=0).stop 2024-09-15T04:02:57.422 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:02:57.423 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:02:57.423 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:02:57.423 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T04:02:57.423 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 28, 2024-09-15T04:02:57.423 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:02:57.423 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:02:57.423 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:02:57.423 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:02:57.423 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.423 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 6, 2024-09-15T04:02:57.423 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:02:57.423 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:02:57.423 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:02:57.424 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:57.424 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:57.424 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:57.424 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:57.424 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:57.424 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:57.424 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:57.424 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:57.424 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:57.424 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.424 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:02:57.424 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:02:57.424 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:57.425 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:57.425 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:57.425 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:57.425 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:57.425 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:57.425 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:57.425 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:57.425 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:57.425 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.425 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.425 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:02:57.425 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:02:57.425 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:02:57.425 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:02:57.426 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:02:57.426 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:02:57.426 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:02:57.426 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:02:57.426 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:02:57.426 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:02:57.426 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:02:57.426 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:02:57.426 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:02:57.426 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:02:57.426 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:02:57.426 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:02:57.427 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:02:57.427 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:57.427 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:57.427 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:57.427 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:57.427 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:57.427 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:57.427 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:57.427 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:57.427 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:57.427 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.427 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:02:57.427 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.427 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:02:57.428 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.428 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:02:57.428 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:02:57.428 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:57.428 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:57.428 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.428 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:57.428 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:02:57.428 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.428 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.428 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.428 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:57.428 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:02:57.428 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:57.429 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.430 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:57.430 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:02:57.430 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.430 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.430 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.430 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:57.430 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:02:57.430 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.430 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.430 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.430 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.430 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:02:57.430 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:02:57.430 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.431 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:57.432 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:02:57.432 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.432 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.432 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.432 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.432 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:02:57.432 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:02:57.432 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:57.432 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:57.432 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:57.432 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.432 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.432 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.432 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:02:57.433 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:02:57.433 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.433 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:57.433 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:57.433 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:02:57.433 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.433 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.433 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:02:57.433 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.433 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:57.433 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:57.433 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 2 2024-09-15T04:02:57.433 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.434 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.434 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.434 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:02:57.434 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:02:57.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe86e8c9640 1 -- 172.21.15.110:0/507495473 >> v2:172.21.15.177:6812/28201 conn(0x7fe854038700 msgr2=0x7fe85403abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe86e8c9640 1 --2- 172.21.15.110:0/507495473 >> v2:172.21.15.177:6812/28201 conn(0x7fe854038700 0x7fe85403abb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe86813de40 tx=0x7fe860009780 comp rx=0 tx=0).stop 2024-09-15T04:02:57.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe86e8c9640 10 monclient: shutdown 2024-09-15T04:02:57.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe86e8c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:57.435 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe86e8c9640 1 -- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 msgr2=0x7fe86813d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe86e8c9640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 0x7fe86813d8c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe85800c8b0 tx=0x7fe858008040 comp rx=0 tx=0).stop 2024-09-15T04:02:57.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe86ce66640 1 -- 172.21.15.110:0/507495473 reap_dead start 2024-09-15T04:02:57.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe86e8c9640 1 -- 172.21.15.110:0/507495473 shutdown_connections 2024-09-15T04:02:57.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe86e8c9640 1 --2- 172.21.15.110:0/507495473 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fe868067130 0x7fe86813d8c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe86e8c9640 1 -- 172.21.15.110:0/507495473 >> 172.21.15.110:0/507495473 conn(0x7fe86809e570 msgr2=0x7fe86806c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:57.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe86e8c9640 1 -- 172.21.15.110:0/507495473 shutdown_connections 2024-09-15T04:02:57.436 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.409+0000 7fe86e8c9640 1 -- 172.21.15.110:0/507495473 wait complete. 2024-09-15T04:02:57.436 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:02:57.436 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T04:02:57.437 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T04:02:57.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.529+0000 7f923b468640 10 monclient: get_monmap_and_config 2024-09-15T04:02:57.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.529+0000 7f923b468640 10 monclient: build_initial_monmap 2024-09-15T04:02:57.530 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.529+0000 7f923b468640 10 monclient: monmap: 2024-09-15T04:02:57.530 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:57.530 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.530 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:02:57.530564+0000 2024-09-15T04:02:57.530 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:02:57.530564+0000 2024-09-15T04:02:57.530 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:57.531 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.531 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:02:57.531 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:02:57.531 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:02:57.531 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:02:57.531 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:02:57.531 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.533 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 1 Processor -- start 2024-09-15T04:02:57.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 1 -- start start 2024-09-15T04:02:57.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 10 monclient: init 2024-09-15T04:02:57.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:57.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-15T04:02:57.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f92340a3410 0x7f923406e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 10 monclient(hunting): picked mon.noname-c con 0x7f92340a3410 addr v2:172.21.15.177:3300/0 2024-09-15T04:02:57.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9234069ff0 0x7f923406ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 10 monclient(hunting): picked mon.noname-b con 0x7f9234069ff0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:57.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f923409d4f0 0x7f923409d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 10 monclient(hunting): picked mon.noname-a con 0x7f923409d4f0 addr v2:172.21.15.110:3300/0 2024-09-15T04:02:57.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92340a37e0 con 0x7f923409d4f0 2024-09-15T04:02:57.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.534 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9234069330 con 0x7f92340a3410 2024-09-15T04:02:57.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9234131fa0 con 0x7f9234069ff0 2024-09-15T04:02:57.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:57.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f923b468640 10 monclient(hunting): authenticate will time out at 1001.174846s 2024-09-15T04:02:57.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.530+0000 7f9239a05640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f923409d4f0 0x7f923409d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f923409d4f0 0x7f923409d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40504/0 (socket says 172.21.15.110:40504) 2024-09-15T04:02:57.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 1 -- 172.21.15.110:0/1504559498 learned_addr learned my addr 172.21.15.110:0/1504559498 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:57.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 10 monclient(hunting): get_auth_request con 0x7f923409d4f0 auth_method 0 2024-09-15T04:02:57.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9238a03640 1 --2- 172.21.15.110:0/1504559498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9234069ff0 0x7f923406ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9238a03640 10 monclient(hunting): get_auth_request con 0x7f9234069ff0 auth_method 0 2024-09-15T04:02:57.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9238a03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9238a03640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9238a03640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239204640 1 --2- 172.21.15.110:0/1504559498 >> v2:172.21.15.177:3300/0 conn(0x7f92340a3410 0x7f923406e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9238a03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9238a03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9238a03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 10 monclient(hunting): handle_auth_done global_id 54747 payload 274 2024-09-15T04:02:57.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 1 -- 172.21.15.110:0/1504559498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9234069ff0 msgr2=0x7f923406ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:57.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 1 --2- 172.21.15.110:0/1504559498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9234069ff0 0x7f923406ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 1 -- 172.21.15.110:0/1504559498 >> v2:172.21.15.177:3300/0 conn(0x7f92340a3410 msgr2=0x7f923406e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:02:57.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 1 --2- 172.21.15.110:0/1504559498 >> v2:172.21.15.177:3300/0 conn(0x7f92340a3410 0x7f923406e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 10 monclient: _finish_hunting 0 2024-09-15T04:02:57.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 1 monclient: found mon.noname-a 2024-09-15T04:02:57.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:57.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:02:57.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239a05640 1 -- 172.21.15.110:0/1504559498 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923409f000 con 0x7f923409d4f0 2024-09-15T04:02:57.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239204640 10 monclient: get_auth_request con 0x7f92340a3410 auth_method 0 2024-09-15T04:02:57.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.531+0000 7f9239204640 1 --2- 172.21.15.110:0/1504559498 >> v2:172.21.15.177:3300/0 conn(0x7f92340a3410 0x7f923406e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:02:57.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f9239a05640 1 -- 172.21.15.110:0/1504559498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f923409d4f0 msgr2=0x7f923409d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:57.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f9239a05640 1 --2- 172.21.15.110:0/1504559498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f923409d4f0 0x7f923409d970 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f923406a420 tx=0x7f922c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d669d549a2bfb02b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f922bfff640 1 -- 172.21.15.110:0/1504559498 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c007610 con 0x7f923409d4f0 2024-09-15T04:02:57.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f922bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f922bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:02:57.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f922bfff640 10 monclient: dump: 2024-09-15T04:02:57.546 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.546 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.546 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.546 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.546 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.547 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.547 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.547 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.547 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.547 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f922bfff640 10 monclient: _finish_auth 0 2024-09-15T04:02:57.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f922bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:57.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f922bfff640 1 -- 172.21.15.110:0/1504559498 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922c02d050 con 0x7f923409d4f0 2024-09-15T04:02:57.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 5 monclient: authenticate success, global_id 54747 2024-09-15T04:02:57.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:02:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f922bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 10 monclient: get_monmap_and_config success 2024-09-15T04:02:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 4 set_mon_vals no callback set 2024-09-15T04:02:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f922bfff640 1 -- 172.21.15.110:0/1504559498 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c0079a0 con 0x7f923409d4f0 2024-09-15T04:02:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f922bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f922bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:02:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f922bfff640 10 monclient: dump: 2024-09-15T04:02:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.548 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.549 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.549 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.549 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.549 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 10 monclient: shutdown 2024-09-15T04:02:57.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:57.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 1 -- 172.21.15.110:0/1504559498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f923409d4f0 msgr2=0x7f923409d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 1 --2- 172.21.15.110:0/1504559498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f923409d4f0 0x7f923409d970 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f923406a420 tx=0x7f922c009840 comp rx=0 tx=0).stop 2024-09-15T04:02:57.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 1 -- 172.21.15.110:0/1504559498 shutdown_connections 2024-09-15T04:02:57.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 1 --2- 172.21.15.110:0/1504559498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f923409d4f0 0x7f923409d970 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 1 --2- 172.21.15.110:0/1504559498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9234069ff0 0x7f923406ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 1 --2- 172.21.15.110:0/1504559498 >> v2:172.21.15.177:3300/0 conn(0x7f92340a3410 0x7f923406e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.549 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 1 -- 172.21.15.110:0/1504559498 >> 172.21.15.110:0/1504559498 conn(0x7f92340635b0 msgr2=0x7f92340a2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:57.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 1 -- 172.21.15.110:0/1504559498 shutdown_connections 2024-09-15T04:02:57.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.534+0000 7f923b468640 1 -- 172.21.15.110:0/1504559498 wait complete. 2024-09-15T04:02:57.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.535+0000 7f923b468640 10 monclient: build_initial_monmap 2024-09-15T04:02:57.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.535+0000 7f923b468640 10 monclient: monmap: 2024-09-15T04:02:57.551 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:02:57.551 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:02:57.551 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:02:57.551 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:02:57.551 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:02:57.551 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.551 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:02:57.551 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:02:57.551 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:02:57.552 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.535+0000 7f923b468640 1 Processor -- start 2024-09-15T04:02:57.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.535+0000 7f923b468640 1 -- start start 2024-09-15T04:02:57.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.535+0000 7f923b468640 10 monclient: init 2024-09-15T04:02:57.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 10 monclient: _reopen_session rank -1 2024-09-15T04:02:57.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T04:02:57.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9234069ff0 0x7f923409bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 10 monclient(hunting): picked mon.noname-a con 0x7f9234069ff0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:02:57.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f923409d4f0 0x7f923412e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 10 monclient(hunting): picked mon.noname-c con 0x7f923409d4f0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:02:57.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92340a3410 0x7f9234119160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 10 monclient(hunting): picked mon.noname-b con 0x7f92340a3410 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:02:57.552 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92340a1360 con 0x7f9234069ff0 2024-09-15T04:02:57.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923411b8c0 con 0x7f92340a3410 2024-09-15T04:02:57.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923411b5e0 con 0x7f923409d4f0 2024-09-15T04:02:57.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 10 monclient(hunting): _renew_subs 2024-09-15T04:02:57.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f923b468640 10 monclient(hunting): authenticate will time out at 1001.180378s 2024-09-15T04:02:57.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239204640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9234069ff0 0x7f923409bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239204640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9234069ff0 0x7f923409bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:40510/0 (socket says 172.21.15.110:40510) 2024-09-15T04:02:57.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239204640 1 -- 172.21.15.110:0/3585036992 learned_addr learned my addr 172.21.15.110:0/3585036992 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:02:57.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239204640 10 monclient(hunting): get_auth_request con 0x7f9234069ff0 auth_method 0 2024-09-15T04:02:57.553 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239204640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239204640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239204640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f923409d4f0 0x7f923412e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 10 monclient(hunting): get_auth_request con 0x7f923409d4f0 auth_method 0 2024-09-15T04:02:57.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239204640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239204640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239204640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.554 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239a05640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92340a3410 0x7f9234119160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239a05640 10 monclient(hunting): get_auth_request con 0x7f92340a3410 auth_method 0 2024-09-15T04:02:57.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239a05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239a05640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239a05640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 10 monclient(hunting): handle_auth_done global_id 64139 payload 824 2024-09-15T04:02:57.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 1 -- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92340a3410 msgr2=0x7f9234119160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f92340a3410 0x7f9234119160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 1 -- 172.21.15.110:0/3585036992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9234069ff0 msgr2=0x7f923409bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9234069ff0 0x7f923409bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.555 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 10 monclient: _finish_hunting 0 2024-09-15T04:02:57.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 1 monclient: found mon.noname-c 2024-09-15T04:02:57.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:02:57.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:02:57.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9238a03640 1 -- 172.21.15.110:0/3585036992 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922c009ae0 con 0x7f923409d4f0 2024-09-15T04:02:57.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.536+0000 7f9239204640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9234069ff0 0x7f923409bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:02:57.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.538+0000 7f9238a03640 1 -- 172.21.15.110:0/3585036992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f923409d4f0 msgr2=0x7f923412e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:57.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.538+0000 7f9238a03640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f923409d4f0 0x7f923412e4c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f922400e930 tx=0x7f92240090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f92297fa640 1 -- 172.21.15.110:0/3585036992 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92240093c0 con 0x7f923409d4f0 2024-09-15T04:02:57.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f92297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f92297fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:02:57.556 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f92297fa640 10 monclient: dump: 2024-09-15T04:02:57.556 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.557 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.557 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.557 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.557 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.557 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.557 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.557 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.557 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.557 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f92297fa640 10 monclient: _finish_auth 0 2024-09-15T04:02:57.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f92297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:02:57.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f92297fa640 1 -- 172.21.15.110:0/3585036992 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922400cb50 con 0x7f923409d4f0 2024-09-15T04:02:57.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f923b468640 5 monclient: authenticate success, global_id 64139 2024-09-15T04:02:57.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f92297fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:02:57.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f92297fa640 1 -- 172.21.15.110:0/3585036992 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9224013470 con 0x7f923409d4f0 2024-09-15T04:02:57.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f923b468640 10 monclient: _renew_subs 2024-09-15T04:02:57.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f923b468640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:02:57.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f923b468640 1 -- 172.21.15.110:0/3585036992 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92341416d0 con 0x7f923409d4f0 2024-09-15T04:02:57.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f92297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f92297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:02:57.558 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f92297fa640 10 monclient: dump: 2024-09-15T04:02:57.558 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.558 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.558 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.559 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.559 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.559 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.559 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.559 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.559 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.559 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f922b7fe640 4 set_mon_vals no callback set 2024-09-15T04:02:57.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f923b468640 10 monclient: _renew_subs 2024-09-15T04:02:57.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f923b468640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:02:57.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f923b468640 1 -- 172.21.15.110:0/3585036992 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9234140f60 con 0x7f923409d4f0 2024-09-15T04:02:57.559 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.539+0000 7f92297fa640 1 -- 172.21.15.110:0/3585036992 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f9224013860 con 0x7f923409d4f0 2024-09-15T04:02:57.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f923b468640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:57.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f923b468640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923411a6e0 0x7f92341196a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f923b468640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f923b468640 1 -- 172.21.15.110:0/3585036992 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9234067ab0 con 0x7f923411a6e0 2024-09-15T04:02:57.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f92297fa640 1 --2- 172.21.15.110:0/3585036992 >> v2:172.21.15.177:6812/28201 conn(0x7f9218038730 0x7f921803abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f92297fa640 1 -- 172.21.15.110:0/3585036992 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f922404b880 con 0x7f923409d4f0 2024-09-15T04:02:57.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f9239a05640 1 --2- 172.21.15.110:0/3585036992 >> v2:172.21.15.177:6812/28201 conn(0x7f9218038730 0x7f921803abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f9239204640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923411a6e0 0x7f92341196a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.560 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f9239a05640 10 monclient: get_auth_request con 0x7f9218038730 auth_method 0 2024-09-15T04:02:57.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f9239204640 10 monclient: get_auth_request con 0x7f923411a6e0 auth_method 0 2024-09-15T04:02:57.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f9239204640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f9239204640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f9239204640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f9239204640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f9239204640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f9239204640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f9239204640 10 monclient(hunting): handle_auth_done global_id 5311 payload 824 2024-09-15T04:02:57.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.540+0000 7f9239204640 1 -- 172.21.15.110:0/3585036992 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92340647e0 con 0x7f923411a6e0 2024-09-15T04:02:57.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.543+0000 7f9239a05640 1 --2- 172.21.15.110:0/3585036992 >> v2:172.21.15.177:6812/28201 conn(0x7f9218038730 0x7f921803abe0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9234069760 tx=0x7f922c036000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.561 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.543+0000 7f9239204640 1 -- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923411a6e0 msgr2=0x7f92341196a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.543+0000 7f9239204640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923411a6e0 0x7f92341196a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f923000af10 tx=0x7f9230008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.543+0000 7f92297fa640 1 -- 172.21.15.110:0/3585036992 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230013650 con 0x7f923411a6e0 2024-09-15T04:02:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.543+0000 7f92297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.543+0000 7f92297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:02:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.543+0000 7f92297fa640 10 monclient: dump: 2024-09-15T04:02:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.562 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.563 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.543+0000 7f92297fa640 1 -- 172.21.15.110:0/3585036992 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f923000c040 con 0x7f923411a6e0 2024-09-15T04:02:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.543+0000 7f92297fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:02:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.543+0000 7f92297fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:02:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.543+0000 7f92297fa640 1 -- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923411a6e0 msgr2=0x7f92341196a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.543+0000 7f92297fa640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923411a6e0 0x7f92341196a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f923000af10 tx=0x7f9230008040 comp rx=0 tx=0).stop 2024-09-15T04:02:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.543+0000 7f92297fa640 1 -- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923411a6e0 msgr2=0x7f92341196a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:02:57.563 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.543+0000 7f92297fa640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923411a6e0 0x7f92341196a0 secure :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f923000af10 tx=0x7f9230008040 comp rx=0 tx=0).stop 2024-09-15T04:02:57.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.549+0000 7f923b468640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:02:57.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.549+0000 7f923b468640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923412ea00 0x7f9234119be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:02:57.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.549+0000 7f923b468640 10 monclient(hunting): start opening mon connection 2024-09-15T04:02:57.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.549+0000 7f923b468640 1 -- 172.21.15.110:0/3585036992 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923411b750 con 0x7f923412ea00 2024-09-15T04:02:57.565 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.549+0000 7f9239204640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923412ea00 0x7f9234119be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:02:57.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.549+0000 7f9239204640 10 monclient: get_auth_request con 0x7f923412ea00 auth_method 0 2024-09-15T04:02:57.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.549+0000 7f9239204640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:02:57.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.549+0000 7f9239204640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:02:57.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.549+0000 7f9239204640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:02:57.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.549+0000 7f9239204640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:02:57.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.549+0000 7f9239204640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:02:57.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.549+0000 7f9239204640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:02:57.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.550+0000 7f9239204640 10 monclient(hunting): handle_auth_done global_id 5314 payload 824 2024-09-15T04:02:57.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.550+0000 7f9239204640 1 -- 172.21.15.110:0/3585036992 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f923406a450 con 0x7f923412ea00 2024-09-15T04:02:57.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.552+0000 7f9239204640 1 -- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923412ea00 msgr2=0x7f9234119be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:02:57.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.552+0000 7f9239204640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923412ea00 0x7f9234119be0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f923000af10 tx=0x7f923001d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:02:57.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.552+0000 7f92297fa640 1 -- 172.21.15.110:0/3585036992 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230023040 con 0x7f923412ea00 2024-09-15T04:02:57.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.552+0000 7f92297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:02:57.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.552+0000 7f92297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:02:57.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.552+0000 7f92297fa640 10 monclient: dump: 2024-09-15T04:02:57.567 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:02:57.567 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:02:57.567 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.567 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:02:57.567 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:02:57.567 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:02:57.568 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:02:57.568 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:02:57.568 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:02:57.568 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:02:57.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f92297fa640 1 -- 172.21.15.110:0/3585036992 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4747 (secure 0 0 0) 0x7f9230006250 con 0x7f923412ea00 2024-09-15T04:02:57.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f92297fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:02:57.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f92297fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:02:57.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f92297fa640 1 -- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923412ea00 msgr2=0x7f9234119be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f92297fa640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923412ea00 0x7f9234119be0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f923000af10 tx=0x7f923001d000 comp rx=0 tx=0).stop 2024-09-15T04:02:57.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f92297fa640 1 -- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923412ea00 msgr2=0x7f9234119be0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:02:57.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f92297fa640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f923412ea00 0x7f9234119be0 secure :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f923000af10 tx=0x7f923001d000 comp rx=0 tx=0).stop 2024-09-15T04:02:57.569 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:02:57.569 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:02:57.569 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:02:57.569 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T04:02:57.569 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 28, 2024-09-15T04:02:57.569 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:02:57.569 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:02:57.569 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:02:57.570 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:02:57.570 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.570 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 6, 2024-09-15T04:02:57.570 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:02:57.570 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:02:57.570 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:02:57.570 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:57.570 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:57.570 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:57.571 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:57.571 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:57.571 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:57.571 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:57.571 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:57.571 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:57.571 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.571 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:02:57.571 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:02:57.571 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:57.572 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:57.572 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:57.572 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:57.572 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:57.572 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:57.572 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:57.572 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:57.572 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:57.572 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.572 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.572 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:02:57.572 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:02:57.572 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:02:57.573 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:02:57.573 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:02:57.573 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:02:57.573 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:02:57.573 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:02:57.573 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:02:57.573 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:02:57.573 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:02:57.573 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:02:57.573 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:02:57.573 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:02:57.574 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:02:57.574 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:02:57.574 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:02:57.574 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:02:57.574 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:02:57.574 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:02:57.574 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:02:57.574 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:02:57.574 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:02:57.574 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:02:57.574 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:02:57.574 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:02:57.574 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.575 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:02:57.575 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.575 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:02:57.575 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.575 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:02:57.575 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:02:57.575 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:57.575 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:57.575 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.575 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:57.575 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:02:57.575 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.576 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.576 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.576 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:57.576 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:02:57.576 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.576 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.576 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.576 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.576 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:02:57.576 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:02:57.576 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:57.577 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:57.577 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:57.577 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.577 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.577 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:02:57.577 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:02:57.577 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:57.577 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:57.577 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.577 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:57.577 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:02:57.578 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.578 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.578 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.578 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:57.578 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:02:57.578 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.578 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.578 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.578 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.578 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:02:57.578 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:02:57.578 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:57.579 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:57.579 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:57.579 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.579 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.579 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:02:57.579 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:02:57.579 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:02:57.579 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:02:57.579 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.579 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:02:57.579 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:02:57.579 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.579 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.579 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.580 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:02:57.580 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:02:57.580 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:02:57.580 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.580 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.580 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.580 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:02:57.580 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:02:57.580 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:02:57.580 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:02:57.580 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:02:57.580 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.581 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.581 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.581 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:02:57.581 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:02:57.581 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.581 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:57.581 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:57.581 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:02:57.581 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.581 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.581 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T04:02:57.582 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.582 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:57.582 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:57.582 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 5 2024-09-15T04:02:57.582 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.582 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.582 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:02:57.582 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.582 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:57.582 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:57.582 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 5 2024-09-15T04:02:57.583 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.583 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:02:57.583 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": [ 2024-09-15T04:02:57.583 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:02:57.583 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:02:57.583 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:02:57.583 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:02:57.583 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:02:57.583 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:02:57.583 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:02:57.583 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:02:57.584 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:02:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f923b468640 1 -- 172.21.15.110:0/3585036992 >> v2:172.21.15.177:6812/28201 conn(0x7f9218038730 msgr2=0x7f921803abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f923b468640 1 --2- 172.21.15.110:0/3585036992 >> v2:172.21.15.177:6812/28201 conn(0x7f9218038730 0x7f921803abe0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9234069760 tx=0x7f922c036000 comp rx=0 tx=0).stop 2024-09-15T04:02:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f923b468640 10 monclient: shutdown 2024-09-15T04:02:57.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f923b468640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:02:57.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f923b468640 1 -- 172.21.15.110:0/3585036992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f923409d4f0 msgr2=0x7f923412e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:02:57.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f923b468640 1 --2- 172.21.15.110:0/3585036992 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f923409d4f0 0x7f923412e4c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f922400e930 tx=0x7f92240090f0 comp rx=0 tx=0).stop 2024-09-15T04:02:57.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f9239a05640 1 -- 172.21.15.110:0/3585036992 reap_dead start 2024-09-15T04:02:57.587 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f9239a05640 1 -- 172.21.15.110:0/3585036992 reap_dead start 2024-09-15T04:02:57.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f923b468640 1 -- 172.21.15.110:0/3585036992 shutdown_connections 2024-09-15T04:02:57.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f923b468640 1 -- 172.21.15.110:0/3585036992 >> 172.21.15.110:0/3585036992 conn(0x7f92340635b0 msgr2=0x7f923406e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:02:57.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f923b468640 1 -- 172.21.15.110:0/3585036992 shutdown_connections 2024-09-15T04:02:57.588 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:02:57.553+0000 7f923b468640 1 -- 172.21.15.110:0/3585036992 wait complete. 2024-09-15T04:02:57.588 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:02:57.589 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T04:02:57.589 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-15T04:02:57.589 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T04:02:57.589 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T04:02:57.589 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T04:02:57.589 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T04:02:57.590 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:57.590 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32593]: starting. 2024-09-15T04:02:57.590 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32593]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32557 2024-09-15T04:02:57.590 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32593]: shutting down. 2024-09-15T04:02:57.590 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:57.591 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:02:57.591 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:02:57.591 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:02:57.598 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T04:02:57.598 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T04:02:57.599 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T04:02:57.599 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T04:03:12.599 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T04:03:12.600 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T04:03:12.600 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T04:03:12.600 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:03:12.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.702+0000 7f439ef5d640 10 monclient: get_monmap_and_config 2024-09-15T04:03:12.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.702+0000 7f439ef5d640 10 monclient: build_initial_monmap 2024-09-15T04:03:12.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.702+0000 7f439ef5d640 10 monclient: monmap: 2024-09-15T04:03:12.703 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:03:12.704 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:12.704 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:03:12.703752+0000 2024-09-15T04:03:12.704 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:03:12.703752+0000 2024-09-15T04:03:12.704 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:03:12.704 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:12.704 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:03:12.704 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:03:12.704 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:03:12.704 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:03:12.704 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:03:12.704 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:12.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.703+0000 7f439ef5d640 1 Processor -- start 2024-09-15T04:03:12.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.703+0000 7f439ef5d640 1 -- start start 2024-09-15T04:03:12.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.703+0000 7f439ef5d640 10 monclient: init 2024-09-15T04:03:12.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ef5d640 10 monclient: _reopen_session rank -1 2024-09-15T04:03:12.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ef5d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T04:03:12.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ef5d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:12.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ef5d640 10 monclient(hunting): picked mon.noname-a con 0x7f439806ead0 addr v2:172.21.15.110:3300/0 2024-09-15T04:03:12.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ef5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f439809b170 addr v1:172.21.15.177:6789/0 2024-09-15T04:03:12.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ef5d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 0x7f4398139330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:12.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ef5d640 10 monclient(hunting): picked mon.noname-b con 0x7f4398067840 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:03:12.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ef5d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f439809a560 con 0x7f439809b170 2024-09-15T04:03:12.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ef5d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:12.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ef5d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4398099fb0 con 0x7f439806ead0 2024-09-15T04:03:12.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ef5d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:12.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ef5d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f439809a250 con 0x7f4398067840 2024-09-15T04:03:12.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ef5d640 10 monclient(hunting): _renew_subs 2024-09-15T04:03:12.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ef5d640 10 monclient(hunting): authenticate will time out at 1016.348137s 2024-09-15T04:03:12.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439d4fa640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 msgr2=0x7f4398139330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:03:12.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439d4fa640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 0x7f4398139330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:03:12.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ccf9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:12.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ccf9640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39818/0 (socket says 172.21.15.110:39818) 2024-09-15T04:03:12.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ccf9640 1 -- 172.21.15.110:0/3319453572 learned_addr learned my addr 172.21.15.110:0/3319453572 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:03:12.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ccf9640 10 monclient(hunting): get_auth_request con 0x7f439806ead0 auth_method 0 2024-09-15T04:03:12.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ccf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:12.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ccf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:12.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ccf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:03:12.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ccf9640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 msgr2=0x7f439809ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T04:03:12.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ccf9640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 msgr2=0x7f439809ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:03:12.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ccf9640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:03:12.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.704+0000 7f439ccf9640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:03:12.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.706+0000 7f4397fff640 1 -- 172.21.15.110:0/3319453572 >> v1:172.21.15.177:6789/0 conn(0x7f439809b170 legacy=0x7f4398067130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:03:12.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.905+0000 7f439d4fa640 1 -- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 msgr2=0x7f4398139330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:03:12.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.905+0000 7f439d4fa640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 0x7f4398139330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T04:03:12.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.905+0000 7f439ccf9640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:12.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.905+0000 7f439ccf9640 10 monclient(hunting): get_auth_request con 0x7f439806ead0 auth_method 0 2024-09-15T04:03:12.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.905+0000 7f439ccf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:12.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.905+0000 7f439ccf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:12.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.905+0000 7f439ccf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:03:12.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.905+0000 7f439ccf9640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 msgr2=0x7f439809ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-15T04:03:12.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.905+0000 7f439ccf9640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 msgr2=0x7f439809ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:03:12.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.905+0000 7f439ccf9640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:03:12.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:12.905+0000 7f439ccf9640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T04:03:13.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:13.306+0000 7f439d4fa640 1 -- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 msgr2=0x7f4398139330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:03:13.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:13.306+0000 7f439d4fa640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 0x7f4398139330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T04:03:13.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:13.306+0000 7f439ccf9640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:13.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:13.306+0000 7f439ccf9640 10 monclient(hunting): get_auth_request con 0x7f439806ead0 auth_method 0 2024-09-15T04:03:13.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:13.306+0000 7f439ccf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:13.306+0000 7f439ccf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:13.306+0000 7f439ccf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:03:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:13.306+0000 7f439ccf9640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 msgr2=0x7f439809ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T04:03:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:13.306+0000 7f439ccf9640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 msgr2=0x7f439809ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:03:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:13.306+0000 7f439ccf9640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:03:13.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:13.306+0000 7f439ccf9640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T04:03:14.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:14.107+0000 7f439d4fa640 1 -- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 msgr2=0x7f4398139330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:03:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:14.107+0000 7f439d4fa640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 0x7f4398139330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T04:03:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:14.107+0000 7f439ccf9640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:14.107+0000 7f439ccf9640 10 monclient(hunting): get_auth_request con 0x7f439806ead0 auth_method 0 2024-09-15T04:03:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:14.107+0000 7f439ccf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:14.107+0000 7f439ccf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:14.107+0000 7f439ccf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:03:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:14.107+0000 7f439ccf9640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 msgr2=0x7f439809ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-15T04:03:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:14.107+0000 7f439ccf9640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 msgr2=0x7f439809ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:03:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:14.107+0000 7f439ccf9640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:03:14.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:14.107+0000 7f439ccf9640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T04:03:15.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 10 monclient(hunting): tick 2024-09-15T04:03:15.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 1 monclient(hunting): continuing hunt 2024-09-15T04:03:15.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T04:03:15.706 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 1 -- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 msgr2=0x7f4398139330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:03:15.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 0x7f4398139330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:15.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 msgr2=0x7f439809ac30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:03:15.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.110:3300/0 conn(0x7f439806ead0 0x7f439809ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:15.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 1 -- 172.21.15.110:0/3319453572 >> v1:172.21.15.177:6789/0 conn(0x7f439809b170 legacy=0x7f4398067130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:15.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 10 monclient: _add_conns ranks=[0,4] 2024-09-15T04:03:15.707 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f437c003720 addr v1:172.21.15.110:6789/0 2024-09-15T04:03:15.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 0x7f437c009850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:15.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 10 monclient(hunting): picked mon.noname-c con 0x7f437c0073a0 addr v2:172.21.15.177:3300/0 2024-09-15T04:03:15.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 1 -- 172.21.15.110:0/3319453572 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f439809d490 con 0x7f437c003720 2024-09-15T04:03:15.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:15.708 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 1 -- 172.21.15.110:0/3319453572 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4398099fb0 con 0x7f437c0073a0 2024-09-15T04:03:15.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f43967fc640 10 monclient(hunting): _renew_subs 2024-09-15T04:03:15.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f4397fff640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 0x7f437c009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:15.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f4397fff640 10 monclient(hunting): get_auth_request con 0x7f437c0073a0 auth_method 0 2024-09-15T04:03:15.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f4397fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:15.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f4397fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:15.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f4397fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:03:15.709 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f4397fff640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 msgr2=0x7f437c009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:03:15.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f4397fff640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 msgr2=0x7f437c009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:03:15.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f4397fff640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 0x7f437c009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:03:15.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.704+0000 7f4397fff640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 0x7f437c009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:03:15.710 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.706+0000 7f439ccf9640 1 -- 172.21.15.110:0/3319453572 >> v1:172.21.15.110:6789/0 conn(0x7f437c003720 legacy=0x7f437c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:03:15.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.905+0000 7f4397fff640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 0x7f437c009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:15.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.905+0000 7f4397fff640 10 monclient(hunting): get_auth_request con 0x7f437c0073a0 auth_method 0 2024-09-15T04:03:15.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.905+0000 7f4397fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:15.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.905+0000 7f4397fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:15.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.905+0000 7f4397fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:03:15.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.905+0000 7f4397fff640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 msgr2=0x7f437c009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:03:15.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.905+0000 7f4397fff640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 msgr2=0x7f437c009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:03:15.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.905+0000 7f4397fff640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 0x7f437c009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:03:15.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:15.905+0000 7f4397fff640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 0x7f437c009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T04:03:16.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:16.306+0000 7f4397fff640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 0x7f437c009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:16.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:16.306+0000 7f4397fff640 10 monclient(hunting): get_auth_request con 0x7f437c0073a0 auth_method 0 2024-09-15T04:03:16.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:16.306+0000 7f4397fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:16.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:16.306+0000 7f4397fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:16.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:16.306+0000 7f4397fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:03:16.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:16.306+0000 7f4397fff640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 msgr2=0x7f437c009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:03:16.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:16.306+0000 7f4397fff640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 msgr2=0x7f437c009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:03:16.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:16.306+0000 7f4397fff640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 0x7f437c009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:03:16.308 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:16.306+0000 7f4397fff640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 0x7f437c009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T04:03:17.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.108+0000 7f4397fff640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 0x7f437c009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:17.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.108+0000 7f4397fff640 10 monclient(hunting): get_auth_request con 0x7f437c0073a0 auth_method 0 2024-09-15T04:03:17.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.108+0000 7f4397fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:17.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.108+0000 7f4397fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:17.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.108+0000 7f4397fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:03:17.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.108+0000 7f4397fff640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 msgr2=0x7f437c009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T04:03:17.109 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.108+0000 7f4397fff640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 msgr2=0x7f437c009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T04:03:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.108+0000 7f4397fff640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 0x7f437c009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:03:17.110 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.108+0000 7f4397fff640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 0x7f437c009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T04:03:17.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.611+0000 7f43977fe640 1 -- 172.21.15.110:0/3319453572 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444496828 0 0) 0x7f439809d490 con 0x7f437c003720 2024-09-15T04:03:17.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.611+0000 7f43977fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:17.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.611+0000 7f43977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:03:17.612 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.611+0000 7f43977fe640 10 monclient(hunting): my global_id is 54765 2024-09-15T04:03:17.613 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.611+0000 7f43977fe640 1 -- 172.21.15.110:0/3319453572 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43880034e0 con 0x7f437c003720 2024-09-15T04:03:17.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 1 -- 172.21.15.110:0/3319453572 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 815671126 0 0) 0x7f43880034e0 con 0x7f437c003720 2024-09-15T04:03:17.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 1 -- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 msgr2=0x7f437c009850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:03:17.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 1 --2- 172.21.15.110:0/3319453572 >> v2:172.21.15.177:3300/0 conn(0x7f437c0073a0 0x7f437c009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient: _finish_hunting 0 2024-09-15T04:03:17.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T04:03:17.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:03:17.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.110:6789/0 2024-09-15T04:03:17.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 1 -- 172.21.15.110:0/3319453572 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c00af60 con 0x7f437c003720 2024-09-15T04:03:17.615 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient: _finish_auth 0 2024-09-15T04:03:17.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:03:17.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 1 -- 172.21.15.110:0/3319453572 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3165669920 0 0) 0x7f438c00c680 con 0x7f437c003720 2024-09-15T04:03:17.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:17.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T04:03:17.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient: dump: 2024-09-15T04:03:17.616 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:17.616 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:17.616 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.616 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.616 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:17.616 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:17.616 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:17.616 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:17.617 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:17.617 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:17.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] but i'm connected to v1:172.21.15.110:6789/0, reconnecting 2024-09-15T04:03:17.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient: _reopen_session rank -1 2024-09-15T04:03:17.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 1 -- 172.21.15.110:0/3319453572 >> v1:172.21.15.110:6789/0 conn(0x7f437c003720 legacy=0x7f437c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:17.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T04:03:17.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f439d4fa640 1 -- 172.21.15.110:0/3319453572 reap_dead start 2024-09-15T04:03:17.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 0x7f43880039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient(hunting): picked mon.c con 0x7f4398067840 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:03:17.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4388004000 0x7f4388004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient(hunting): picked mon.b con 0x7f4388004000 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:03:17.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004cc0 0x7f4388005130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient(hunting): picked mon.a con 0x7f4388004cc0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:03:17.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 1 -- 172.21.15.110:0/3319453572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4398099fb0 con 0x7f4388004cc0 2024-09-15T04:03:17.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 1 -- 172.21.15.110:0/3319453572 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438800bbc0 con 0x7f4388004000 2024-09-15T04:03:17.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 1 -- 172.21.15.110:0/3319453572 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438800bd80 con 0x7f4398067840 2024-09-15T04:03:17.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient(hunting): _renew_subs 2024-09-15T04:03:17.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f43977fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T04:03:17.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f439ccf9640 1 -- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 msgr2=0x7f43880039d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:03:17.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f439ccf9640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 0x7f43880039d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f4397fff640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4388004000 0x7f4388004780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f439d4fa640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004cc0 0x7f4388005130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f4397fff640 10 monclient(hunting): get_auth_request con 0x7f4388004000 auth_method 0 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f4397fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f4397fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f4397fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f439d4fa640 10 monclient(hunting): get_auth_request con 0x7f4388004cc0 auth_method 0 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f439d4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f439d4fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.612+0000 7f439d4fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 10 monclient(hunting): handle_auth_done global_id 54765 payload 293 2024-09-15T04:03:17.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 1 -- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 msgr2=0x7f43880039d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:03:17.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 0x7f43880039d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 1 -- 172.21.15.110:0/3319453572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4388004000 msgr2=0x7f4388004780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:03:17.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4388004000 0x7f4388004780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 10 monclient: _finish_hunting 0 2024-09-15T04:03:17.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 1 monclient: found mon.a 2024-09-15T04:03:17.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:03:17.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:03:17.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 1 -- 172.21.15.110:0/3319453572 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43880034e0 con 0x7f4388004cc0 2024-09-15T04:03:17.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 10 monclient: _finish_auth 0 2024-09-15T04:03:17.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439d4fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:03:17.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439ef5d640 5 monclient: authenticate success, global_id 54765 2024-09-15T04:03:17.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.613+0000 7f439ef5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:03:17.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.616+0000 7f439d4fa640 1 -- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004cc0 msgr2=0x7f4388005130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:03:17.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.616+0000 7f439d4fa640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004cc0 0x7f4388005130 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f439809d230 tx=0x7f4390007580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a347edcd817d9a94 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:03:17.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.617+0000 7f43977fe640 1 -- 172.21.15.110:0/3319453572 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4390008040 con 0x7f4388004cc0 2024-09-15T04:03:17.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.617+0000 7f43977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:17.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.617+0000 7f43977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:03:17.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.617+0000 7f43977fe640 10 monclient: dump: 2024-09-15T04:03:17.627 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:17.627 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:17.627 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.628 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.628 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:17.628 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:17.628 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:17.628 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:17.628 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:17.628 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:17.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.617+0000 7f43977fe640 1 -- 172.21.15.110:0/3319453572 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439002ac70 con 0x7f4388004cc0 2024-09-15T04:03:17.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.617+0000 7f43977fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:03:17.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.617+0000 7f43977fe640 1 -- 172.21.15.110:0/3319453572 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4390032410 con 0x7f4388004cc0 2024-09-15T04:03:17.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.617+0000 7f439ef5d640 10 monclient: get_monmap_and_config success 2024-09-15T04:03:17.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.617+0000 7f439ef5d640 4 set_mon_vals no callback set 2024-09-15T04:03:17.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.617+0000 7f43977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:17.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.617+0000 7f43977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:03:17.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.617+0000 7f43977fe640 10 monclient: dump: 2024-09-15T04:03:17.629 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:17.629 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:17.629 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.629 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.629 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:17.629 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:17.629 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:17.629 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:17.629 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:17.629 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:17.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.618+0000 7f439ef5d640 10 monclient: shutdown 2024-09-15T04:03:17.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.618+0000 7f439ef5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:03:17.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.618+0000 7f439ef5d640 1 -- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004cc0 msgr2=0x7f4388005130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.618+0000 7f439ef5d640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004cc0 0x7f4388005130 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f439809d230 tx=0x7f4390007580 comp rx=0 tx=0).stop 2024-09-15T04:03:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.618+0000 7f439ef5d640 1 -- 172.21.15.110:0/3319453572 shutdown_connections 2024-09-15T04:03:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.618+0000 7f439ef5d640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004cc0 0x7f4388005130 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.618+0000 7f439ef5d640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4388004000 0x7f4388004780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.618+0000 7f439ef5d640 1 --2- 172.21.15.110:0/3319453572 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4398067840 0x7f43880039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.618+0000 7f439ef5d640 1 -- 172.21.15.110:0/3319453572 >> 172.21.15.110:0/3319453572 conn(0x7f439809e5e0 msgr2=0x7f439809e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:03:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.618+0000 7f439ef5d640 1 -- 172.21.15.110:0/3319453572 shutdown_connections 2024-09-15T04:03:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.618+0000 7f439ef5d640 1 -- 172.21.15.110:0/3319453572 wait complete. 2024-09-15T04:03:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.619+0000 7f439ef5d640 10 monclient: build_initial_monmap 2024-09-15T04:03:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.619+0000 7f439ef5d640 10 monclient: monmap: 2024-09-15T04:03:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:03:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:03:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:03:17.630 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:03:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:03:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:03:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:03:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:03:17.631 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.620+0000 7f439ef5d640 1 Processor -- start 2024-09-15T04:03:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 1 -- start start 2024-09-15T04:03:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 10 monclient: init 2024-09-15T04:03:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 10 monclient: _reopen_session rank -1 2024-09-15T04:03:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T04:03:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004000 0x7f4398140a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 10 monclient(hunting): picked mon.noname-a con 0x7f4388004000 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:03:17.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4388004cc0 0x7f4398133610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 10 monclient(hunting): picked mon.noname-c con 0x7f4388004cc0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:03:17.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4398067840 0x7f439811d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 10 monclient(hunting): picked mon.noname-b con 0x7f4398067840 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:03:17.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4398120730 con 0x7f4388004000 2024-09-15T04:03:17.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43981343b0 con 0x7f4398067840 2024-09-15T04:03:17.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f439806e8a0 con 0x7f4388004cc0 2024-09-15T04:03:17.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 10 monclient(hunting): _renew_subs 2024-09-15T04:03:17.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ef5d640 10 monclient(hunting): authenticate will time out at 1021.265798s 2024-09-15T04:03:17.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f4397fff640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4388004cc0 msgr2=0x7f4398133610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:03:17.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f4397fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4388004cc0 0x7f4398133610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:03:17.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ccf9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004000 0x7f4398140a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:17.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439d4fa640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4398067840 0x7f439811d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:17.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.621+0000 7f439ccf9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004000 0x7f4398140a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39854/0 (socket says 172.21.15.110:39854) 2024-09-15T04:03:17.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 1 -- 172.21.15.110:0/1011154236 learned_addr learned my addr 172.21.15.110:0/1011154236 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:03:17.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 10 monclient(hunting): get_auth_request con 0x7f4388004000 auth_method 0 2024-09-15T04:03:17.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:17.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:17.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:03:17.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439d4fa640 10 monclient(hunting): get_auth_request con 0x7f4398067840 auth_method 0 2024-09-15T04:03:17.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439d4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:17.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439d4fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:17.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439d4fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:03:17.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:03:17.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:03:17.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:03:17.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439d4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:03:17.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439d4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:03:17.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439d4fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:03:17.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 10 monclient(hunting): handle_auth_done global_id 54768 payload 824 2024-09-15T04:03:17.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 1 -- 172.21.15.110:0/1011154236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4388004cc0 msgr2=0x7f4398133610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:03:17.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 1 --2- 172.21.15.110:0/1011154236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4388004cc0 0x7f4398133610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 1 -- 172.21.15.110:0/1011154236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4398067840 msgr2=0x7f439811d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:17.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 1 --2- 172.21.15.110:0/1011154236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4398067840 0x7f439811d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 10 monclient: _finish_hunting 0 2024-09-15T04:03:17.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 1 monclient: found mon.noname-a 2024-09-15T04:03:17.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:03:17.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:03:17.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439ccf9640 1 -- 172.21.15.110:0/1011154236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439811f110 con 0x7f4388004000 2024-09-15T04:03:17.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.622+0000 7f439d4fa640 1 --2- 172.21.15.110:0/1011154236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4398067840 0x7f439811d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:03:17.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.624+0000 7f439ccf9640 1 -- 172.21.15.110:0/1011154236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004000 msgr2=0x7f4398140a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:03:17.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.624+0000 7f439ccf9640 1 --2- 172.21.15.110:0/1011154236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004000 0x7f4398140a00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f438c014940 tx=0x7f438c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:03:17.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f4394ff9640 1 -- 172.21.15.110:0/1011154236 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c014ed0 con 0x7f4388004000 2024-09-15T04:03:17.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f4394ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:17.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f4394ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:03:17.635 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f4394ff9640 10 monclient: dump: 2024-09-15T04:03:17.635 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:17.635 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:17.635 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.635 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.635 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:17.635 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:17.636 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:17.636 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:17.636 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:17.636 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:17.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f4394ff9640 10 monclient: _finish_auth 0 2024-09-15T04:03:17.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f4394ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:03:17.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f439ef5d640 5 monclient: authenticate success, global_id 54768 2024-09-15T04:03:17.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f439ef5d640 10 monclient: _renew_subs 2024-09-15T04:03:17.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f439ef5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:03:17.636 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f439ef5d640 1 -- 172.21.15.110:0/1011154236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4398141850 con 0x7f4388004000 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f439ef5d640 10 monclient: _renew_subs 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f439ef5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f439ef5d640 1 -- 172.21.15.110:0/1011154236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4398140f40 con 0x7f4388004000 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f4394ff9640 1 -- 172.21.15.110:0/1011154236 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438c00c040 con 0x7f4388004000 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f4394ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f4394ff9640 1 -- 172.21.15.110:0/1011154236 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c012de0 con 0x7f4388004000 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f4394ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f4394ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f4394ff9640 10 monclient: dump: 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:17.637 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:17.638 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:17.638 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:17.638 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:17.638 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:17.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.625+0000 7f4396ffd640 4 set_mon_vals no callback set 2024-09-15T04:03:17.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.626+0000 7f4394ff9640 1 -- 172.21.15.110:0/1011154236 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f438c017680 con 0x7f4388004000 2024-09-15T04:03:17.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.626+0000 7f439ef5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:03:17.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.626+0000 7f439ef5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:03:17.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.626+0000 7f439ef5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:03:17.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.626+0000 7f439ef5d640 1 -- 172.21.15.110:0/1011154236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4378005010 con 0x7f4388004000 2024-09-15T04:03:17.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.627+0000 7f4394ff9640 1 --2- 172.21.15.110:0/1011154236 >> v2:172.21.15.177:6812/28201 conn(0x7f437c038f30 0x7f437c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.627+0000 7f4394ff9640 1 -- 172.21.15.110:0/1011154236 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f438c04c000 con 0x7f4388004000 2024-09-15T04:03:17.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.627+0000 7f4397fff640 1 --2- 172.21.15.110:0/1011154236 >> v2:172.21.15.177:6812/28201 conn(0x7f437c038f30 0x7f437c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:17.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.627+0000 7f4397fff640 10 monclient: get_auth_request con 0x7f437c038f30 auth_method 0 2024-09-15T04:03:17.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.629+0000 7f4397fff640 1 --2- 172.21.15.110:0/1011154236 >> v2:172.21.15.177:6812/28201 conn(0x7f437c038f30 0x7f437c03b3e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4384007ad0 tx=0x7f4384009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:03:17.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.632+0000 7f4394ff9640 1 -- 172.21.15.110:0/1011154236 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f438c015dd0 con 0x7f4388004000 2024-09-15T04:03:17.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.632+0000 7f4394ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:03:17.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.632+0000 7f4394ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:03:17.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.849+0000 7f439ef5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:03:17.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.849+0000 7f439ef5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:03:17.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.849+0000 7f439ef5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:03:17.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.849+0000 7f439ef5d640 1 -- 172.21.15.110:0/1011154236 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4378005840 con 0x7f4388004000 2024-09-15T04:03:17.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.850+0000 7f4394ff9640 1 -- 172.21.15.110:0/1011154236 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1533 (secure 0 0 0) 0x7f438c0157d0 con 0x7f4388004000 2024-09-15T04:03:17.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.850+0000 7f4394ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:03:17.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.850+0000 7f4394ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:03:17.851 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:03:17.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.851+0000 7f439ef5d640 1 -- 172.21.15.110:0/1011154236 >> v2:172.21.15.177:6812/28201 conn(0x7f437c038f30 msgr2=0x7f437c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:17.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.851+0000 7f439ef5d640 1 --2- 172.21.15.110:0/1011154236 >> v2:172.21.15.177:6812/28201 conn(0x7f437c038f30 0x7f437c03b3e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4384007ad0 tx=0x7f4384009040 comp rx=0 tx=0).stop 2024-09-15T04:03:17.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.851+0000 7f439ef5d640 10 monclient: shutdown 2024-09-15T04:03:17.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.851+0000 7f439ef5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:03:17.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.851+0000 7f439ef5d640 1 -- 172.21.15.110:0/1011154236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004000 msgr2=0x7f4398140a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:17.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.851+0000 7f439ef5d640 1 --2- 172.21.15.110:0/1011154236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004000 0x7f4398140a00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f438c014940 tx=0x7f438c008040 comp rx=0 tx=0).stop 2024-09-15T04:03:17.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.851+0000 7f439ef5d640 1 -- 172.21.15.110:0/1011154236 shutdown_connections 2024-09-15T04:03:17.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.851+0000 7f439ef5d640 1 --2- 172.21.15.110:0/1011154236 >> v2:172.21.15.177:6812/28201 conn(0x7f437c038f30 0x7f437c03b3e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.851+0000 7f439ef5d640 1 --2- 172.21.15.110:0/1011154236 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4398067840 0x7f439811d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.851+0000 7f439ef5d640 1 --2- 172.21.15.110:0/1011154236 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4388004cc0 0x7f4398133610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.851+0000 7f439ef5d640 1 --2- 172.21.15.110:0/1011154236 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4388004000 0x7f4398140a00 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.851+0000 7f439ef5d640 1 -- 172.21.15.110:0/1011154236 >> 172.21.15.110:0/1011154236 conn(0x7f439809e5e0 msgr2=0x7f43981345c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:03:17.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.851+0000 7f439ef5d640 1 -- 172.21.15.110:0/1011154236 shutdown_connections 2024-09-15T04:03:17.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.851+0000 7f439ef5d640 1 -- 172.21.15.110:0/1011154236 wait complete. 2024-09-15T04:03:17.863 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T04:03:17.863 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T04:03:17.863 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T04:03:17.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.955+0000 7ffaef11d640 10 monclient: get_monmap_and_config 2024-09-15T04:03:17.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.955+0000 7ffaef11d640 10 monclient: build_initial_monmap 2024-09-15T04:03:17.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.955+0000 7ffaef11d640 10 monclient: monmap: 2024-09-15T04:03:17.956 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:03:17.956 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:17.957 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:03:17.956582+0000 2024-09-15T04:03:17.957 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:03:17.956582+0000 2024-09-15T04:03:17.957 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:03:17.957 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:17.957 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:03:17.957 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:03:17.957 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:03:17.957 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:03:17.957 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:03:17.957 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 1 Processor -- start 2024-09-15T04:03:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 1 -- start start 2024-09-15T04:03:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 10 monclient: init 2024-09-15T04:03:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 10 monclient: _reopen_session rank -1 2024-09-15T04:03:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T04:03:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae809b0d0 0x7ffae806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 10 monclient(hunting): picked mon.noname-b con 0x7ffae809b0d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:03:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffae8067130 addr v1:172.21.15.177:6789/0 2024-09-15T04:03:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffae8067c10 addr v1:172.21.15.110:6789/0 2024-09-15T04:03:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae80a1330 con 0x7ffae8067c10 2024-09-15T04:03:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae809a290 con 0x7ffae8067130 2024-09-15T04:03:17.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae8099fb0 con 0x7ffae809b0d0 2024-09-15T04:03:17.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 10 monclient(hunting): _renew_subs 2024-09-15T04:03:17.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaef11d640 10 monclient(hunting): authenticate will time out at 1021.600818s 2024-09-15T04:03:17.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaeceb9640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae809b0d0 msgr2=0x7ffae806e8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:03:17.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaeceb9640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae809b0d0 0x7ffae806e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:03:17.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaed6ba640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ffae8067c10 0x7ffae81392c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:60616/0 (socket says 172.21.15.110:60616) 2024-09-15T04:03:17.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.956+0000 7ffaed6ba640 1 -- 172.21.15.110:0/2705619653 learned_addr learned my addr 172.21.15.110:0/2705619653 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:03:17.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaed6ba640 1 -- 172.21.15.110:0/2705619653 >> v1:172.21.15.110:6789/0 conn(0x7ffae8067c10 legacy=0x7ffae81392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:03:17.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 -- 172.21.15.110:0/2705619653 <== mon.0 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251965143 0 0) 0x7ffae80a1330 con 0x7ffae8067c10 2024-09-15T04:03:17.967 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:03:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient(hunting): my global_id is 54771 2024-09-15T04:03:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 -- 172.21.15.110:0/2705619653 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad00034e0 con 0x7ffae8067c10 2024-09-15T04:03:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 -- 172.21.15.110:0/2705619653 <== mon.0 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1210236290 0 0) 0x7ffad00034e0 con 0x7ffae8067c10 2024-09-15T04:03:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 -- 172.21.15.110:0/2705619653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae809b0d0 msgr2=0x7ffae806e8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:03:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 --2- 172.21.15.110:0/2705619653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae809b0d0 0x7ffae806e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 -- 172.21.15.110:0/2705619653 >> v1:172.21.15.177:6789/0 conn(0x7ffae8067130 legacy=0x7ffae8067500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadffff640 1 -- 172.21.15.110:0/2705619653 >> v1:172.21.15.177:6789/0 conn(0x7ffae8067130 legacy=0x7ffae8067500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:03:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient: _finish_hunting 0 2024-09-15T04:03:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 monclient: found mon.noname-a-legacy 2024-09-15T04:03:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:03:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.110:6789/0 2024-09-15T04:03:17.968 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 -- 172.21.15.110:0/2705619653 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae809a570 con 0x7ffae8067c10 2024-09-15T04:03:17.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient: _finish_auth 0 2024-09-15T04:03:17.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:03:17.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 -- 172.21.15.110:0/2705619653 <== mon.0 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3165669920 0 0) 0x7ffae0001aa0 con 0x7ffae8067c10 2024-09-15T04:03:17.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaef11d640 5 monclient: authenticate success, global_id 54771 2024-09-15T04:03:17.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaef11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:03:17.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:17.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-15T04:03:17.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient: dump: 2024-09-15T04:03:17.969 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:17.969 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:17.969 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.969 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.969 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:17.969 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] but i'm connected to v1:172.21.15.110:6789/0, reconnecting 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient: _reopen_session rank -1 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 -- 172.21.15.110:0/2705619653 >> v1:172.21.15.110:6789/0 conn(0x7ffae8067c10 legacy=0x7ffae81392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 --2- 172.21.15.110:0/2705619653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad0003a80 0x7ffad0003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient(hunting): picked mon.b con 0x7ffad0003a80 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 --2- 172.21.15.110:0/2705619653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae809b0d0 0x7ffad0004780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient(hunting): picked mon.c con 0x7ffae809b0d0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 --2- 172.21.15.110:0/2705619653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0004e70 0x7ffad000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient(hunting): picked mon.a con 0x7ffad0004e70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:03:17.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 -- 172.21.15.110:0/2705619653 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae8099fb0 con 0x7ffad0004e70 2024-09-15T04:03:17.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 -- 172.21.15.110:0/2705619653 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad0005620 con 0x7ffad0003a80 2024-09-15T04:03:17.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 1 -- 172.21.15.110:0/2705619653 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad000bf60 con 0x7ffae809b0d0 2024-09-15T04:03:17.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient(hunting): _renew_subs 2024-09-15T04:03:17.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T04:03:17.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadffff640 1 -- 172.21.15.110:0/2705619653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae809b0d0 msgr2=0x7ffad0004780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:03:17.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaef11d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-15T04:03:17.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffadffff640 1 --2- 172.21.15.110:0/2705619653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae809b0d0 0x7ffad0004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:03:17.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaed6ba640 1 --2- 172.21.15.110:0/2705619653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0004e70 0x7ffad000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:17.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaeceb9640 1 --2- 172.21.15.110:0/2705619653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad0003a80 0x7ffad0003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:17.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaed6ba640 10 monclient(hunting): get_auth_request con 0x7ffad0004e70 auth_method 0 2024-09-15T04:03:17.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaed6ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:17.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaed6ba640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:17.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaed6ba640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:03:17.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaeceb9640 10 monclient(hunting): get_auth_request con 0x7ffad0003a80 auth_method 0 2024-09-15T04:03:17.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaeceb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:17.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaeceb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:17.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaeceb9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:03:17.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaed6ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:03:17.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaed6ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:03:17.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaed6ba640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:03:17.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaeceb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:03:17.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaeceb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:03:17.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaeceb9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:03:17.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.959+0000 7ffaed6ba640 10 monclient(hunting): handle_auth_done global_id 54771 payload 293 2024-09-15T04:03:17.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.960+0000 7ffaed6ba640 1 -- 172.21.15.110:0/2705619653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae809b0d0 msgr2=0x7ffad0004780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:03:17.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.960+0000 7ffaed6ba640 1 --2- 172.21.15.110:0/2705619653 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae809b0d0 0x7ffad0004780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.960+0000 7ffaed6ba640 1 -- 172.21.15.110:0/2705619653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad0003a80 msgr2=0x7ffad0003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:03:17.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.960+0000 7ffaed6ba640 1 --2- 172.21.15.110:0/2705619653 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffad0003a80 0x7ffad0003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.960+0000 7ffaed6ba640 10 monclient: _finish_hunting 0 2024-09-15T04:03:17.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.960+0000 7ffaed6ba640 1 monclient: found mon.a 2024-09-15T04:03:17.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.960+0000 7ffaed6ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:03:17.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.960+0000 7ffaed6ba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:03:17.973 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.960+0000 7ffaed6ba640 1 -- 172.21.15.110:0/2705619653 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00034e0 con 0x7ffad0004e70 2024-09-15T04:03:17.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.960+0000 7ffaed6ba640 10 monclient: _finish_auth 0 2024-09-15T04:03:17.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.960+0000 7ffaed6ba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:03:17.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.962+0000 7ffaed6ba640 1 -- 172.21.15.110:0/2705619653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0004e70 msgr2=0x7ffad000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:03:17.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.962+0000 7ffaed6ba640 1 --2- 172.21.15.110:0/2705619653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0004e70 0x7ffad000b9d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ffae0001810 tx=0x7ffae809ad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c5376d4e61d1b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:03:17.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.962+0000 7ffaed6ba640 1 --2- 172.21.15.110:0/2705619653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0004e70 0x7ffad000b9d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ffae0001810 tx=0x7ffae809ad00 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7ffae002b620 mon_map magic: 0 v1 2024-09-15T04:03:17.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.962+0000 7ffadf7fe640 1 -- 172.21.15.110:0/2705619653 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae002b620 con 0x7ffad0004e70 2024-09-15T04:03:17.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.962+0000 7ffadf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:17.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.962+0000 7ffadf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:03:17.974 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.962+0000 7ffadf7fe640 10 monclient: dump: 2024-09-15T04:03:17.974 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:17.974 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:17.974 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.974 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.975 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:17.975 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:17.975 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:17.975 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:17.975 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:17.975 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:17.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.962+0000 7ffadf7fe640 1 -- 172.21.15.110:0/2705619653 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae002b780 con 0x7ffad0004e70 2024-09-15T04:03:17.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.962+0000 7ffaef11d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:03:17.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.962+0000 7ffadf7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:03:17.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.962+0000 7ffaef11d640 10 monclient: get_monmap_and_config success 2024-09-15T04:03:17.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.962+0000 7ffaef11d640 4 set_mon_vals no callback set 2024-09-15T04:03:17.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.963+0000 7ffaef11d640 10 monclient: shutdown 2024-09-15T04:03:17.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.963+0000 7ffaef11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:03:17.975 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.963+0000 7ffaef11d640 1 -- 172.21.15.110:0/2705619653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0004e70 msgr2=0x7ffad000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:17.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.963+0000 7ffaef11d640 1 --2- 172.21.15.110:0/2705619653 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0004e70 0x7ffad000b9d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ffae0001810 tx=0x7ffae809ad00 comp rx=0 tx=0).stop 2024-09-15T04:03:17.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.963+0000 7ffaed6ba640 1 -- 172.21.15.110:0/2705619653 reap_dead start 2024-09-15T04:03:17.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.963+0000 7ffadf7fe640 1 -- 172.21.15.110:0/2705619653 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7ffae002bb40 con 0x7ffad0004e70 2024-09-15T04:03:17.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.963+0000 7ffadf7fe640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T04:03:17.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.963+0000 7ffaef11d640 1 -- 172.21.15.110:0/2705619653 shutdown_connections 2024-09-15T04:03:17.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.963+0000 7ffaef11d640 1 -- 172.21.15.110:0/2705619653 >> 172.21.15.110:0/2705619653 conn(0x7ffae809e570 msgr2=0x7ffae809e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:03:17.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.963+0000 7ffaef11d640 1 -- 172.21.15.110:0/2705619653 shutdown_connections 2024-09-15T04:03:17.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.963+0000 7ffaef11d640 1 -- 172.21.15.110:0/2705619653 wait complete. 2024-09-15T04:03:17.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.964+0000 7ffaef11d640 10 monclient: build_initial_monmap 2024-09-15T04:03:17.976 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.964+0000 7ffaef11d640 10 monclient: monmap: 2024-09-15T04:03:17.976 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:03:17.976 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:03:17.976 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:03:17.977 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:03:17.977 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:03:17.977 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:17.977 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:03:17.977 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:03:17.977 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:03:17.977 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:17.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.964+0000 7ffaef11d640 1 Processor -- start 2024-09-15T04:03:17.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.964+0000 7ffaef11d640 1 -- start start 2024-09-15T04:03:17.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.964+0000 7ffaef11d640 10 monclient: init 2024-09-15T04:03:17.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.964+0000 7ffaef11d640 10 monclient: _reopen_session rank -1 2024-09-15T04:03:17.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.964+0000 7ffaef11d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T04:03:17.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.964+0000 7ffaef11d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0004e70 0x7ffae813d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.977 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.964+0000 7ffaef11d640 10 monclient(hunting): picked mon.noname-a con 0x7ffad0004e70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:03:17.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.964+0000 7ffaef11d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae811fb50 0x7ffae813cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.964+0000 7ffaef11d640 10 monclient(hunting): picked mon.noname-c con 0x7ffae811fb50 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:03:17.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.964+0000 7ffaef11d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffae8121020 0x7ffae8140d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaef11d640 10 monclient(hunting): picked mon.noname-b con 0x7ffae8121020 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:03:17.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaef11d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaef11d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae81204e0 con 0x7ffad0004e70 2024-09-15T04:03:17.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaef11d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaef11d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae80a1690 con 0x7ffae8121020 2024-09-15T04:03:17.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaef11d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaef11d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae806e830 con 0x7ffae811fb50 2024-09-15T04:03:17.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaef11d640 10 monclient(hunting): _renew_subs 2024-09-15T04:03:17.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaef11d640 10 monclient(hunting): authenticate will time out at 1021.609028s 2024-09-15T04:03:17.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffadffff640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae811fb50 msgr2=0x7ffae813cea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:03:17.978 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffadffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae811fb50 0x7ffae813cea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:03:17.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0004e70 0x7ffae813d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:17.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0004e70 0x7ffae813d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39872/0 (socket says 172.21.15.110:39872) 2024-09-15T04:03:17.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 1 -- 172.21.15.110:0/185168454 learned_addr learned my addr 172.21.15.110:0/185168454 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:03:17.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 10 monclient(hunting): get_auth_request con 0x7ffad0004e70 auth_method 0 2024-09-15T04:03:17.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:17.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:17.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:03:17.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:03:17.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:03:17.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:03:17.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 10 monclient(hunting): handle_auth_done global_id 54774 payload 824 2024-09-15T04:03:17.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 1 -- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae811fb50 msgr2=0x7ffae813cea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:03:17.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 1 --2- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ffae811fb50 0x7ffae813cea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.979 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 1 -- 172.21.15.110:0/185168454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffae8121020 msgr2=0x7ffae8140d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:17.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 1 --2- 172.21.15.110:0/185168454 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffae8121020 0x7ffae8140d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 10 monclient: _finish_hunting 0 2024-09-15T04:03:17.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 1 monclient: found mon.noname-a 2024-09-15T04:03:17.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:03:17.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:03:17.980 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.965+0000 7ffaeceb9640 1 -- 172.21.15.110:0/185168454 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae8121f40 con 0x7ffad0004e70 2024-09-15T04:03:17.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffaeceb9640 1 -- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0004e70 msgr2=0x7ffae813d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:03:17.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffaeceb9640 1 --2- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0004e70 0x7ffae813d7f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ffae4006cf0 tx=0x7ffae4009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:03:17.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffadcff9640 1 -- 172.21.15.110:0/185168454 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae400e900 con 0x7ffad0004e70 2024-09-15T04:03:17.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffadcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:17.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffadcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:03:17.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffadcff9640 10 monclient: dump: 2024-09-15T04:03:17.985 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:17.985 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:17.985 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.985 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.985 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:17.985 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:17.985 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:17.985 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:17.985 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:17.985 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffadcff9640 10 monclient: _finish_auth 0 2024-09-15T04:03:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffadcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:03:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffadcff9640 1 -- 172.21.15.110:0/185168454 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae400ea80 con 0x7ffad0004e70 2024-09-15T04:03:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffaef11d640 5 monclient: authenticate success, global_id 54774 2024-09-15T04:03:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffaef11d640 10 monclient: _renew_subs 2024-09-15T04:03:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffaef11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:03:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffaef11d640 1 -- 172.21.15.110:0/185168454 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae8122220 con 0x7ffad0004e70 2024-09-15T04:03:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffadcff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:03:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffadcff9640 1 -- 172.21.15.110:0/185168454 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae400ee40 con 0x7ffad0004e70 2024-09-15T04:03:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffadcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffadcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:03:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffadcff9640 10 monclient: dump: 2024-09-15T04:03:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:17.986 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:17.987 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.987 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.987 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:17.987 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:17.987 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:17.987 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:17.987 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:17.987 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:17.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffaef11d640 10 monclient: _renew_subs 2024-09-15T04:03:17.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffaef11d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:03:17.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffadeffd640 4 set_mon_vals no callback set 2024-09-15T04:03:17.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.967+0000 7ffaef11d640 1 -- 172.21.15.110:0/185168454 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae813d580 con 0x7ffad0004e70 2024-09-15T04:03:17.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffaef11d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:03:17.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffaef11d640 1 --2- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae80298c0 0x7ffae8029c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffaef11d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffaef11d640 1 -- 172.21.15.110:0/185168454 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae8068c10 con 0x7ffae80298c0 2024-09-15T04:03:17.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffadcff9640 1 -- 172.21.15.110:0/185168454 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7ffae4012600 con 0x7ffad0004e70 2024-09-15T04:03:17.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffadffff640 1 --2- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae80298c0 0x7ffae8029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:17.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffadffff640 10 monclient: get_auth_request con 0x7ffae80298c0 auth_method 0 2024-09-15T04:03:17.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffadffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:17.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffadffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:17.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffadffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:03:17.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffadffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:03:17.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffadffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:03:17.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffadffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:03:17.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffadcff9640 1 --2- 172.21.15.110:0/185168454 >> v2:172.21.15.177:6812/28201 conn(0x7ffad40386b0 0x7ffad403ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffadffff640 10 monclient(hunting): handle_auth_done global_id 54777 payload 824 2024-09-15T04:03:17.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffadffff640 1 -- 172.21.15.110:0/185168454 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffae80647e0 con 0x7ffae80298c0 2024-09-15T04:03:17.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.968+0000 7ffadcff9640 1 -- 172.21.15.110:0/185168454 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 3894+0+0 (secure 0 0 0) 0x7ffae404b600 con 0x7ffad0004e70 2024-09-15T04:03:17.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.969+0000 7ffaed6ba640 1 --2- 172.21.15.110:0/185168454 >> v2:172.21.15.177:6812/28201 conn(0x7ffad40386b0 0x7ffad403ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:17.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.969+0000 7ffaed6ba640 10 monclient: get_auth_request con 0x7ffad40386b0 auth_method 0 2024-09-15T04:03:17.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.971+0000 7ffadffff640 1 -- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae80298c0 msgr2=0x7ffae8029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:03:17.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.971+0000 7ffadffff640 1 --2- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae80298c0 0x7ffae8029c90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ffad80014b0 tx=0x7ffad8007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:03:17.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.971+0000 7ffadcff9640 1 -- 172.21.15.110:0/185168454 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad80075d0 con 0x7ffae80298c0 2024-09-15T04:03:17.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.971+0000 7ffadcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:17.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.971+0000 7ffadcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:03:17.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.971+0000 7ffadcff9640 10 monclient: dump: 2024-09-15T04:03:17.989 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:17.989 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:17.989 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.990 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.990 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:17.990 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:17.990 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:17.990 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:17.990 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:17.990 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:17.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.971+0000 7ffadcff9640 1 -- 172.21.15.110:0/185168454 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7ffad8005780 con 0x7ffae80298c0 2024-09-15T04:03:17.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.971+0000 7ffadcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:03:17.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.971+0000 7ffadcff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:03:17.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.971+0000 7ffadcff9640 1 -- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae80298c0 msgr2=0x7ffae8029c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:17.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.971+0000 7ffadcff9640 1 --2- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae80298c0 0x7ffae8029c90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ffad80014b0 tx=0x7ffad8007cb0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.971+0000 7ffadcff9640 1 -- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae80298c0 msgr2=0x7ffae8029c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:03:17.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.971+0000 7ffadcff9640 1 --2- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae80298c0 0x7ffae8029c90 secure :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ffad80014b0 tx=0x7ffad8007cb0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.971+0000 7ffaed6ba640 1 --2- 172.21.15.110:0/185168454 >> v2:172.21.15.177:6812/28201 conn(0x7ffad40386b0 0x7ffad403ab60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffae0033000 tx=0x7ffae0031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:03:17.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.977+0000 7ffaef11d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:03:17.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.977+0000 7ffaef11d640 1 --2- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae802a1d0 0x7ffae802a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:17.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.977+0000 7ffaef11d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:17.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.977+0000 7ffaef11d640 1 -- 172.21.15.110:0/185168454 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae80a0760 con 0x7ffae802a1d0 2024-09-15T04:03:17.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.977+0000 7ffadffff640 1 --2- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae802a1d0 0x7ffae802a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:17.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.977+0000 7ffadffff640 10 monclient: get_auth_request con 0x7ffae802a1d0 auth_method 0 2024-09-15T04:03:17.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.977+0000 7ffadffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:17.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.977+0000 7ffadffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:17.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.977+0000 7ffadffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:03:17.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.977+0000 7ffadffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:03:17.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.977+0000 7ffadffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:03:17.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.977+0000 7ffadffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:03:17.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.977+0000 7ffadffff640 10 monclient(hunting): handle_auth_done global_id 54780 payload 824 2024-09-15T04:03:17.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.977+0000 7ffadffff640 1 -- 172.21.15.110:0/185168454 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7ffae8122590 con 0x7ffae802a1d0 2024-09-15T04:03:17.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.979+0000 7ffadffff640 1 -- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae802a1d0 msgr2=0x7ffae802a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:03:17.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.979+0000 7ffadffff640 1 --2- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae802a1d0 0x7ffae802a5d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ffad8014000 tx=0x7ffad8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:03:17.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.979+0000 7ffadcff9640 1 -- 172.21.15.110:0/185168454 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad8020620 con 0x7ffae802a1d0 2024-09-15T04:03:17.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.979+0000 7ffadcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:17.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.979+0000 7ffadcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:03:17.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.979+0000 7ffadcff9640 10 monclient: dump: 2024-09-15T04:03:17.992 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:17.992 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:17.992 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.992 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:17.992 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:17.993 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:17.993 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:17.993 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:17.993 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:17.993 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:17.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.979+0000 7ffadcff9640 1 -- 172.21.15.110:0/185168454 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4568 (secure 0 0 0) 0x7ffad801f370 con 0x7ffae802a1d0 2024-09-15T04:03:17.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.979+0000 7ffadcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:03:17.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.979+0000 7ffadcff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:03:17.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.979+0000 7ffadcff9640 1 -- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae802a1d0 msgr2=0x7ffae802a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:17.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.979+0000 7ffadcff9640 1 --2- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae802a1d0 0x7ffae802a5d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ffad8014000 tx=0x7ffad8015000 comp rx=0 tx=0).stop 2024-09-15T04:03:17.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.979+0000 7ffadcff9640 1 -- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae802a1d0 msgr2=0x7ffae802a5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:03:17.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.979+0000 7ffadcff9640 1 --2- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffae802a1d0 0x7ffae802a5d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:17.994 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:03:17.994 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:03:17.994 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:03:17.994 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "leader", 2024-09-15T04:03:17.994 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 30, 2024-09-15T04:03:17.994 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:03:17.994 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:03:17.994 INFO:teuthology.orchestra.run.smithi110.stdout: 1 2024-09-15T04:03:17.995 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:03:17.995 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 0, 2024-09-15T04:03:17.995 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:03:17.995 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:03:17.995 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:03:17.995 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:03:17.995 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:03:17.995 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:03:17.995 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:03:17.995 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:03:17.995 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:03:17.995 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:03:17.996 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:03:17.996 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:03:17.996 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:03:17.996 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:03:17.996 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:03:17.996 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:03:17.996 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:03:17.996 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:03:17.996 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:03:17.996 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:03:17.996 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:03:17.996 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:03:17.996 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:03:17.996 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:03:17.997 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:03:17.997 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:17.997 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:03:17.997 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:03:17.997 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:03:17.997 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:03:17.997 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:03:17.997 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:03:17.997 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:03:17.997 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:03:17.997 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:03:17.997 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:03:17.998 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:03:17.998 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:03:17.998 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:03:17.998 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:03:17.998 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:03:17.998 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:03:17.998 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:03:17.998 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:03:17.998 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:03:17.998 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:03:17.998 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:03:17.998 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:03:17.998 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:03:17.998 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:03:17.999 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:03:17.999 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:03:17.999 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:03:17.999 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:03:17.999 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:17.999 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:03:17.999 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:17.999 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:03:17.999 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:03:17.999 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:03:17.999 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:03:17.999 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:17.999 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:03:17.999 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:03:18.000 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:03:18.000 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.000 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.000 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:03:18.000 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:03:18.000 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:03:18.000 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.000 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:03:18.000 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.000 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:03:18.000 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:03:18.000 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:03:18.000 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:03:18.000 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:03:18.001 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.001 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.001 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:03:18.001 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:03:18.001 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:03:18.001 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:03:18.001 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.001 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:03:18.001 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:03:18.001 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:03:18.001 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.001 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.001 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:03:18.001 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:03:18.002 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:03:18.002 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.002 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:03:18.002 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.002 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:03:18.002 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:03:18.002 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:03:18.002 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:03:18.002 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:03:18.002 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.002 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.002 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:03:18.002 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:03:18.002 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:03:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:03:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:03:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:03:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:03:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:03:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:03:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:03:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:03:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.003 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:03:18.004 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:03:18.004 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:03:18.004 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:03:18.004 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:03:18.004 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.004 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:03:18.004 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.004 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:03:18.004 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:03:18.004 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.004 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:03:18.004 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:03:18.004 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:03:18.005 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.005 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:03:18.005 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T04:03:18.005 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.005 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:03:18.005 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:03:18.005 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:03:18.005 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.005 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:03:18.005 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:03:18.005 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.005 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:03:18.005 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:03:18.005 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 4 2024-09-15T04:03:18.006 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.006 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:03:18.006 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.006 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:03:18.006 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:03:18.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.980+0000 7ffaef11d640 1 -- 172.21.15.110:0/185168454 >> v2:172.21.15.177:6812/28201 conn(0x7ffad40386b0 msgr2=0x7ffad403ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:18.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.980+0000 7ffaef11d640 1 --2- 172.21.15.110:0/185168454 >> v2:172.21.15.177:6812/28201 conn(0x7ffad40386b0 0x7ffad403ab60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffae0033000 tx=0x7ffae0031000 comp rx=0 tx=0).stop 2024-09-15T04:03:18.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.980+0000 7ffaef11d640 10 monclient: shutdown 2024-09-15T04:03:18.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.980+0000 7ffaef11d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:03:18.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.980+0000 7ffaef11d640 1 -- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0004e70 msgr2=0x7ffae813d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:18.007 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.980+0000 7ffaef11d640 1 --2- 172.21.15.110:0/185168454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffad0004e70 0x7ffae813d7f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ffae4006cf0 tx=0x7ffae4009060 comp rx=0 tx=0).stop 2024-09-15T04:03:18.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.980+0000 7ffaed6ba640 1 -- 172.21.15.110:0/185168454 reap_dead start 2024-09-15T04:03:18.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.980+0000 7ffaed6ba640 1 -- 172.21.15.110:0/185168454 reap_dead start 2024-09-15T04:03:18.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.980+0000 7ffaef11d640 1 -- 172.21.15.110:0/185168454 shutdown_connections 2024-09-15T04:03:18.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.980+0000 7ffaef11d640 1 -- 172.21.15.110:0/185168454 >> 172.21.15.110:0/185168454 conn(0x7ffae809e570 msgr2=0x7ffae806cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:03:18.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.980+0000 7ffaef11d640 1 -- 172.21.15.110:0/185168454 shutdown_connections 2024-09-15T04:03:18.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:17.980+0000 7ffaef11d640 1 -- 172.21.15.110:0/185168454 wait complete. 2024-09-15T04:03:18.008 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T04:03:18.008 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T04:03:18.008 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T04:03:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.101+0000 7f3ac7a6f640 10 monclient: get_monmap_and_config 2024-09-15T04:03:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.101+0000 7f3ac7a6f640 10 monclient: build_initial_monmap 2024-09-15T04:03:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.101+0000 7f3ac7a6f640 10 monclient: monmap: 2024-09-15T04:03:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:03:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:03:18.102594+0000 2024-09-15T04:03:18.102 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:03:18.102594+0000 2024-09-15T04:03:18.103 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:03:18.103 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:18.103 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:03:18.103 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:03:18.103 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:03:18.103 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:03:18.103 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:03:18.103 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:18.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 1 Processor -- start 2024-09-15T04:03:18.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 1 -- start start 2024-09-15T04:03:18.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 10 monclient: init 2024-09-15T04:03:18.105 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 10 monclient: _reopen_session rank -1 2024-09-15T04:03:18.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-15T04:03:18.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f3ac00a3480 0x7f3ac006e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:18.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 10 monclient(hunting): picked mon.noname-c con 0x7f3ac00a3480 addr v2:172.21.15.177:3300/0 2024-09-15T04:03:18.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ac012f5e0 0x7f3ac012fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:18.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 10 monclient(hunting): picked mon.noname-b con 0x7f3ac012f5e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:03:18.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3ac012ff60 addr v1:172.21.15.177:6789/0 2024-09-15T04:03:18.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac009f290 con 0x7f3ac012ff60 2024-09-15T04:03:18.106 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:18.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac00a3850 con 0x7f3ac00a3480 2024-09-15T04:03:18.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:18.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac01303e0 con 0x7f3ac012f5e0 2024-09-15T04:03:18.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 10 monclient(hunting): _renew_subs 2024-09-15T04:03:18.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac7a6f640 10 monclient(hunting): authenticate will time out at 1021.746863s 2024-09-15T04:03:18.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac500a640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ac012f5e0 msgr2=0x7f3ac012fa20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:03:18.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.102+0000 7f3ac500a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ac012f5e0 0x7f3ac012fa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:03:18.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.103+0000 7f3ac600c640 1 --1- >> v1:172.21.15.177:6789/0 conn(0x7f3ac012ff60 0x7f3ac009d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.177:6789/0 says I am v1:172.21.15.110:44766/0 (socket says 172.21.15.110:44766) 2024-09-15T04:03:18.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.103+0000 7f3ac600c640 1 -- 172.21.15.110:0/2142844982 learned_addr learned my addr 172.21.15.110:0/2142844982 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:03:18.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.103+0000 7f3ac580b640 1 --2- 172.21.15.110:0/2142844982 >> v2:172.21.15.177:3300/0 conn(0x7f3ac00a3480 0x7f3ac006e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:18.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.103+0000 7f3ac580b640 10 monclient(hunting): get_auth_request con 0x7f3ac00a3480 auth_method 0 2024-09-15T04:03:18.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.103+0000 7f3ac580b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:18.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.103+0000 7f3ac580b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:18.107 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.103+0000 7f3ac580b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:03:18.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.103+0000 7f3ac580b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:03:18.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.103+0000 7f3ac580b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:03:18.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.103+0000 7f3ac580b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:03:18.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.104+0000 7f3ac580b640 10 monclient(hunting): handle_auth_done global_id 5320 payload 274 2024-09-15T04:03:18.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.104+0000 7f3ac580b640 1 -- 172.21.15.110:0/2142844982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ac012f5e0 msgr2=0x7f3ac012fa20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T04:03:18.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.104+0000 7f3ac580b640 1 --2- 172.21.15.110:0/2142844982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ac012f5e0 0x7f3ac012fa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:18.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.104+0000 7f3ac580b640 1 -- 172.21.15.110:0/2142844982 >> v1:172.21.15.177:6789/0 conn(0x7f3ac012ff60 legacy=0x7f3ac009d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:03:18.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.104+0000 7f3ac580b640 10 monclient: _finish_hunting 0 2024-09-15T04:03:18.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.104+0000 7f3ac580b640 1 monclient: found mon.noname-c 2024-09-15T04:03:18.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.104+0000 7f3ac580b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:03:18.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.104+0000 7f3ac580b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T04:03:18.108 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.104+0000 7f3ac580b640 1 -- 172.21.15.110:0/2142844982 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac009f050 con 0x7f3ac00a3480 2024-09-15T04:03:18.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.107+0000 7f3ac580b640 1 -- 172.21.15.110:0/2142844982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac00a3480 msgr2=0x7f3ac006e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:03:18.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.107+0000 7f3ac580b640 1 --2- 172.21.15.110:0/2142844982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac00a3480 0x7f3ac006e8d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009850 tx=0x7f3abc0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63985382bcc599a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:03:18.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac4809640 1 -- 172.21.15.110:0/2142844982 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc032440 con 0x7f3ac00a3480 2024-09-15T04:03:18.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac4809640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:18.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac4809640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:03:18.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac4809640 10 monclient: dump: 2024-09-15T04:03:18.120 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:18.120 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:18.120 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:18.120 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:18.120 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:18.120 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:18.120 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:18.120 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac4809640 10 monclient: _finish_auth 0 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac4809640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac4809640 1 -- 172.21.15.110:0/2142844982 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3abc0325a0 con 0x7f3ac00a3480 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac7a6f640 5 monclient: authenticate success, global_id 5320 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac7a6f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac4809640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac7a6f640 10 monclient: get_monmap_and_config success 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac4809640 1 -- 172.21.15.110:0/2142844982 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc032940 con 0x7f3ac00a3480 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac4809640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac7a6f640 4 set_mon_vals no callback set 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac4809640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac4809640 10 monclient: dump: 2024-09-15T04:03:18.121 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:18.122 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:18.122 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:18.122 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:18.122 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:18.122 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:18.122 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:18.122 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:18.122 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:18.122 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:18.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac7a6f640 10 monclient: shutdown 2024-09-15T04:03:18.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac7a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:03:18.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/2142844982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac00a3480 msgr2=0x7f3ac006e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:18.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac7a6f640 1 --2- 172.21.15.110:0/2142844982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac00a3480 0x7f3ac006e8d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009850 tx=0x7f3abc0090f0 comp rx=0 tx=0).stop 2024-09-15T04:03:18.122 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/2142844982 shutdown_connections 2024-09-15T04:03:18.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac7a6f640 1 --2- 172.21.15.110:0/2142844982 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ac012f5e0 0x7f3ac012fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:18.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac7a6f640 1 --2- 172.21.15.110:0/2142844982 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac00a3480 0x7f3ac006e8d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:18.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/2142844982 >> 172.21.15.110:0/2142844982 conn(0x7f3ac00635b0 msgr2=0x7f3ac00a2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:03:18.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/2142844982 shutdown_connections 2024-09-15T04:03:18.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.108+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/2142844982 wait complete. 2024-09-15T04:03:18.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.109+0000 7f3ac7a6f640 10 monclient: build_initial_monmap 2024-09-15T04:03:18.123 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.109+0000 7f3ac7a6f640 10 monclient: monmap: 2024-09-15T04:03:18.123 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:03:18.123 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:03:18.123 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:03:18.123 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:03:18.123 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:03:18.123 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:18.123 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:03:18.124 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:03:18.124 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:03:18.124 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:18.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.109+0000 7f3ac7a6f640 1 Processor -- start 2024-09-15T04:03:18.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.109+0000 7f3ac7a6f640 1 -- start start 2024-09-15T04:03:18.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.109+0000 7f3ac7a6f640 10 monclient: init 2024-09-15T04:03:18.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.109+0000 7f3ac7a6f640 10 monclient: _reopen_session rank -1 2024-09-15T04:03:18.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.109+0000 7f3ac7a6f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T04:03:18.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac7a6f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ac00a3480 0x7f3ac013ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:18.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac7a6f640 10 monclient(hunting): picked mon.noname-c con 0x7f3ac00a3480 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:03:18.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac7a6f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ac012f5e0 0x7f3ac012c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:18.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac7a6f640 10 monclient(hunting): picked mon.noname-a con 0x7f3ac012f5e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:03:18.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac7a6f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac012ff60 0x7f3ac0099ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:18.124 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac7a6f640 10 monclient(hunting): picked mon.noname-b con 0x7f3ac012ff60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:03:18.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac7a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:18.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac7a6f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac006c8e0 con 0x7f3ac012f5e0 2024-09-15T04:03:18.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac7a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:18.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac7a6f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac0132540 con 0x7f3ac012ff60 2024-09-15T04:03:18.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac7a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:18.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac7a6f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac009bb20 con 0x7f3ac00a3480 2024-09-15T04:03:18.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac7a6f640 10 monclient(hunting): _renew_subs 2024-09-15T04:03:18.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac580b640 1 -- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ac00a3480 msgr2=0x7f3ac013ea90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.110:3301/0 2024-09-15T04:03:18.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac7a6f640 10 monclient(hunting): authenticate will time out at 1021.754066s 2024-09-15T04:03:18.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac580b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ac00a3480 0x7f3ac013ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T04:03:18.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ac012f5e0 0x7f3ac012c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:18.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ac012f5e0 0x7f3ac012c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:39910/0 (socket says 172.21.15.110:39910) 2024-09-15T04:03:18.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 1 -- 172.21.15.110:0/3134565200 learned_addr learned my addr 172.21.15.110:0/3134565200 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:03:18.125 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 10 monclient(hunting): get_auth_request con 0x7f3ac012f5e0 auth_method 0 2024-09-15T04:03:18.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:18.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:18.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:03:18.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:03:18.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:03:18.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:03:18.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac600c640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac012ff60 0x7f3ac0099ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:18.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 10 monclient(hunting): handle_auth_done global_id 54783 payload 824 2024-09-15T04:03:18.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 1 -- 172.21.15.110:0/3134565200 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ac00a3480 msgr2=0x7f3ac013ea90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T04:03:18.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f3ac00a3480 0x7f3ac013ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:18.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 1 -- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac012ff60 msgr2=0x7f3ac0099ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:18.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac012ff60 0x7f3ac0099ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:18.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 10 monclient: _finish_hunting 0 2024-09-15T04:03:18.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 1 monclient: found mon.noname-a 2024-09-15T04:03:18.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:03:18.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:03:18.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac500a640 1 -- 172.21.15.110:0/3134565200 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3abc009390 con 0x7f3ac012f5e0 2024-09-15T04:03:18.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac600c640 10 monclient: get_auth_request con 0x7f3ac012ff60 auth_method 0 2024-09-15T04:03:18.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.110+0000 7f3ac600c640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac012ff60 0x7f3ac0099ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:03:18.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ac500a640 1 -- 172.21.15.110:0/3134565200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ac012f5e0 msgr2=0x7f3ac012c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:03:18.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ac500a640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ac012f5e0 0x7f3ac012c680 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000b5c0 tx=0x7f3ab0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:03:18.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ab5ffb640 1 -- 172.21.15.110:0/3134565200 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0003d90 con 0x7f3ac012f5e0 2024-09-15T04:03:18.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ab5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:18.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ab5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:03:18.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ab5ffb640 10 monclient: dump: 2024-09-15T04:03:18.128 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:18.128 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:18.128 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:18.128 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:18.128 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:18.128 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:18.128 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:18.128 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:18.128 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:18.128 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:18.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ab5ffb640 10 monclient: _finish_auth 0 2024-09-15T04:03:18.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ab5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:03:18.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ab5ffb640 1 -- 172.21.15.110:0/3134565200 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab0006150 con 0x7f3ac012f5e0 2024-09-15T04:03:18.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ac7a6f640 5 monclient: authenticate success, global_id 54783 2024-09-15T04:03:18.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ac7a6f640 10 monclient: _renew_subs 2024-09-15T04:03:18.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ac7a6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:03:18.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/3134565200 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac009b670 con 0x7f3ac012f5e0 2024-09-15T04:03:18.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ab5ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:03:18.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ab5ffb640 1 -- 172.21.15.110:0/3134565200 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab0006510 con 0x7f3ac012f5e0 2024-09-15T04:03:18.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ab5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:18.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ab5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:03:18.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ab5ffb640 10 monclient: dump: 2024-09-15T04:03:18.129 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:18.129 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:18.129 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:18.129 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:18.129 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.113+0000 7f3ac7a6f640 10 monclient: _renew_subs 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.112+0000 7f3ab7fff640 4 set_mon_vals no callback set 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.113+0000 7f3ac7a6f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.113+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/3134565200 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac009c1e0 con 0x7f3ac012f5e0 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.113+0000 7f3ac7a6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.113+0000 7f3ab5ffb640 1 -- 172.21.15.110:0/3134565200 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f3ab00103c0 con 0x7f3ac012f5e0 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.113+0000 7f3ac7a6f640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac012cbc0 0x7f3ac009a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.113+0000 7f3ac7a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.113+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/3134565200 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac0067ab0 con 0x7f3ac012cbc0 2024-09-15T04:03:18.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.114+0000 7f3ab5ffb640 1 --2- 172.21.15.110:0/3134565200 >> v2:172.21.15.177:6812/28201 conn(0x7f3aa40386b0 0x7f3aa403ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:18.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.114+0000 7f3ab5ffb640 1 -- 172.21.15.110:0/3134565200 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 3894+0+0 (secure 0 0 0) 0x7f3ab004afa0 con 0x7f3ac012f5e0 2024-09-15T04:03:18.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.114+0000 7f3ac580b640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac012cbc0 0x7f3ac009a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:18.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.114+0000 7f3ac580b640 10 monclient: get_auth_request con 0x7f3ac012cbc0 auth_method 0 2024-09-15T04:03:18.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.114+0000 7f3ac580b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:18.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.114+0000 7f3ac580b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:18.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.114+0000 7f3ac580b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:03:18.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.114+0000 7f3ac600c640 1 --2- 172.21.15.110:0/3134565200 >> v2:172.21.15.177:6812/28201 conn(0x7f3aa40386b0 0x7f3aa403ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:18.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.114+0000 7f3ac580b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:03:18.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.114+0000 7f3ac580b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:03:18.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.114+0000 7f3ac580b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:03:18.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.114+0000 7f3ac600c640 10 monclient: get_auth_request con 0x7f3aa40386b0 auth_method 0 2024-09-15T04:03:18.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.114+0000 7f3ac580b640 10 monclient(hunting): handle_auth_done global_id 5323 payload 824 2024-09-15T04:03:18.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.114+0000 7f3ac580b640 1 -- 172.21.15.110:0/3134565200 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ac00647e0 con 0x7f3ac012cbc0 2024-09-15T04:03:18.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.117+0000 7f3ac580b640 1 -- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac012cbc0 msgr2=0x7f3ac009a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:03:18.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.117+0000 7f3ac580b640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac012cbc0 0x7f3ac009a010 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3ac009ecb0 tx=0x7f3abc032000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:03:18.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.117+0000 7f3ac600c640 1 --2- 172.21.15.110:0/3134565200 >> v2:172.21.15.177:6812/28201 conn(0x7f3aa40386b0 0x7f3aa403ab60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8005b90 tx=0x7f3ab8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:03:18.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.117+0000 7f3ab5ffb640 1 -- 172.21.15.110:0/3134565200 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc032440 con 0x7f3ac012cbc0 2024-09-15T04:03:18.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.117+0000 7f3ab5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:18.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.117+0000 7f3ab5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:03:18.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.117+0000 7f3ab5ffb640 10 monclient: dump: 2024-09-15T04:03:18.132 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:18.132 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:18.132 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:18.132 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:18.132 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:18.132 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:18.132 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:18.132 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:18.133 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:18.133 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:18.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.117+0000 7f3ab5ffb640 1 -- 172.21.15.110:0/3134565200 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f3abc031e90 con 0x7f3ac012cbc0 2024-09-15T04:03:18.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.117+0000 7f3ab5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:03:18.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.117+0000 7f3ab5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:03:18.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.117+0000 7f3ab5ffb640 1 -- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac012cbc0 msgr2=0x7f3ac009a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:18.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.117+0000 7f3ab5ffb640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac012cbc0 0x7f3ac009a010 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3ac009ecb0 tx=0x7f3abc032000 comp rx=0 tx=0).stop 2024-09-15T04:03:18.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.117+0000 7f3ab5ffb640 1 -- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac012cbc0 msgr2=0x7f3ac009a010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:03:18.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.117+0000 7f3ab5ffb640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac012cbc0 0x7f3ac009a010 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:03:18.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.123+0000 7f3ac7a6f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:03:18.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.123+0000 7f3ac7a6f640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac009a550 0x7f3ac0095a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:03:18.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.123+0000 7f3ac7a6f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:03:18.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.123+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/3134565200 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac0119ad0 con 0x7f3ac009a550 2024-09-15T04:03:18.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.123+0000 7f3ac580b640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac009a550 0x7f3ac0095a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:03:18.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.123+0000 7f3ac580b640 10 monclient: get_auth_request con 0x7f3ac009a550 auth_method 0 2024-09-15T04:03:18.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.123+0000 7f3ac580b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:03:18.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.123+0000 7f3ac580b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:03:18.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.123+0000 7f3ac580b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:03:18.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.123+0000 7f3ac580b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:03:18.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.123+0000 7f3ac580b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:03:18.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.123+0000 7f3ac580b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:03:18.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.123+0000 7f3ac580b640 10 monclient(hunting): handle_auth_done global_id 5326 payload 824 2024-09-15T04:03:18.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.123+0000 7f3ac580b640 1 -- 172.21.15.110:0/3134565200 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3ac009a920 con 0x7f3ac009a550 2024-09-15T04:03:18.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ac580b640 1 -- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac009a550 msgr2=0x7f3ac0095a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:03:18.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ac580b640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac009a550 0x7f3ac0095a70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009130 tx=0x7f3abc03d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:03:18.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ab5ffb640 1 -- 172.21.15.110:0/3134565200 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3abc046900 con 0x7f3ac009a550 2024-09-15T04:03:18.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ab5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:03:18.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ab5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:03:18.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ab5ffb640 10 monclient: dump: 2024-09-15T04:03:18.136 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:03:18.137 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:03:18.137 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:18.137 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:03:18.137 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:03:18.137 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:03:18.137 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:03:18.137 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:03:18.137 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:03:18.137 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:03:18.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ab5ffb640 1 -- 172.21.15.110:0/3134565200 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4736 (secure 0 0 0) 0x7f3abc000bc0 con 0x7f3ac009a550 2024-09-15T04:03:18.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ab5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:03:18.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ab5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:03:18.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ab5ffb640 1 -- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac009a550 msgr2=0x7f3ac0095a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:18.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ab5ffb640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac009a550 0x7f3ac0095a70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009130 tx=0x7f3abc03d000 comp rx=0 tx=0).stop 2024-09-15T04:03:18.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ab5ffb640 1 -- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac009a550 msgr2=0x7f3ac0095a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:03:18.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ab5ffb640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3ac009a550 0x7f3ac0095a70 secure :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009130 tx=0x7f3abc03d000 comp rx=0 tx=0).stop 2024-09-15T04:03:18.138 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:03:18.138 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:03:18.138 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:03:18.138 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T04:03:18.138 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 30, 2024-09-15T04:03:18.138 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:03:18.138 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:03:18.138 INFO:teuthology.orchestra.run.smithi110.stdout: 1 2024-09-15T04:03:18.138 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:03:18.139 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 0, 2024-09-15T04:03:18.139 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:03:18.139 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:03:18.139 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:03:18.139 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:03:18.139 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:03:18.139 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:03:18.139 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:03:18.139 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:03:18.139 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:03:18.139 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:03:18.139 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:03:18.139 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:03:18.139 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:03:18.140 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:03:18.140 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:03:18.140 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:03:18.140 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:03:18.140 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:03:18.140 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:03:18.140 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:03:18.140 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:03:18.140 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:03:18.140 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:03:18.140 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:03:18.140 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:03:18.140 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.140 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:03:18.141 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:03:18.141 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:03:18.141 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:03:18.141 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:03:18.141 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:03:18.141 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:03:18.141 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:03:18.141 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:03:18.141 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:03:18.141 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:03:18.141 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:03:18.141 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:03:18.141 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:03:18.141 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.142 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:03:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:03:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:03:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:03:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:03:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:03:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:03:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:03:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:03:18.143 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:03:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:03:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:03:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:03:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:03:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:03:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:03:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:03:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:03:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:03:18.144 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:03:18.145 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:03:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:03:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:03:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:03:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:03:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:03:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:03:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:03:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:03:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:03:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:03:18.146 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:03:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:03:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:03:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:03:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:03:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:03:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:03:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:03:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:03:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.147 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:03:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:03:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:03:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:03:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:03:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:03:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:03:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T04:03:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:03:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:03:18.148 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 5 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 5 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": [ 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:03:18.149 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:03:18.150 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:03:18.150 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:03:18.150 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:03:18.150 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:03:18.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/3134565200 >> v2:172.21.15.177:6812/28201 conn(0x7f3aa40386b0 msgr2=0x7f3aa403ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:18.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ac7a6f640 1 --2- 172.21.15.110:0/3134565200 >> v2:172.21.15.177:6812/28201 conn(0x7f3aa40386b0 0x7f3aa403ab60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8005b90 tx=0x7f3ab8009040 comp rx=0 tx=0).stop 2024-09-15T04:03:18.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ac7a6f640 10 monclient: shutdown 2024-09-15T04:03:18.151 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ac7a6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:03:18.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/3134565200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ac012f5e0 msgr2=0x7f3ac012c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:03:18.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ac7a6f640 1 --2- 172.21.15.110:0/3134565200 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ac012f5e0 0x7f3ac012c680 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3ab000b5c0 tx=0x7f3ab0009120 comp rx=0 tx=0).stop 2024-09-15T04:03:18.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ac600c640 1 -- 172.21.15.110:0/3134565200 reap_dead start 2024-09-15T04:03:18.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ac600c640 1 -- 172.21.15.110:0/3134565200 reap_dead start 2024-09-15T04:03:18.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/3134565200 shutdown_connections 2024-09-15T04:03:18.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/3134565200 >> 172.21.15.110:0/3134565200 conn(0x7f3ac00635b0 msgr2=0x7f3ac0132750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:03:18.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/3134565200 shutdown_connections 2024-09-15T04:03:18.152 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:03:18.126+0000 7f3ac7a6f640 1 -- 172.21.15.110:0/3134565200 wait complete. 2024-09-15T04:03:18.152 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T04:03:18.152 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T04:03:19.315 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32661]: starting. 2024-09-15T04:03:19.315 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32661]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32660 2024-09-15T04:03:19.315 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32661]: created object 0... 2024-09-15T04:03:19.315 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32661]: created object 25... 2024-09-15T04:03:19.315 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32661]: created object 49... 2024-09-15T04:03:19.316 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32661]: finishing. 2024-09-15T04:03:19.316 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32661]: shutting down. 2024-09-15T04:03:20.235 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32662]: starting. 2024-09-15T04:03:20.236 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32662]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32660 2024-09-15T04:03:20.236 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32662]: shutting down. 2024-09-15T04:03:21.878 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:21.878 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32695]: starting. 2024-09-15T04:03:21.878 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32695]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32660 2024-09-15T04:03:21.878 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32695]: created object 0... 2024-09-15T04:03:21.878 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32695]: created object 25... 2024-09-15T04:03:21.878 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32695]: created object 49... 2024-09-15T04:03:21.878 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32695]: finishing. 2024-09-15T04:03:21.878 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32695]: shutting down. 2024-09-15T04:03:21.916 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:21.916 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32697]: starting. 2024-09-15T04:03:21.916 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32697]: listing objects. 2024-09-15T04:03:21.916 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32697]: listed object 0... 2024-09-15T04:03:21.917 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32697]: listed object 25... 2024-09-15T04:03:21.917 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32697]: saw 50 objects 2024-09-15T04:03:21.917 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32697]: shutting down. 2024-09-15T04:03:22.797 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:22.797 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32696]: starting. 2024-09-15T04:03:22.798 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32696]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32660 2024-09-15T04:03:22.798 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32696]: shutting down. 2024-09-15T04:03:22.798 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:22.798 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:22.799 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:03:22.801 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:03:24.913 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32764]: starting. 2024-09-15T04:03:24.913 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32764]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32763 2024-09-15T04:03:24.913 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32764]: created object 0... 2024-09-15T04:03:24.913 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32764]: created object 25... 2024-09-15T04:03:24.913 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32764]: created object 49... 2024-09-15T04:03:24.913 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32764]: finishing. 2024-09-15T04:03:24.913 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32764]: shutting down. 2024-09-15T04:03:25.807 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32765]: starting. 2024-09-15T04:03:25.807 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32765]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32763 2024-09-15T04:03:25.807 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32765]: shutting down. 2024-09-15T04:03:27.922 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:27.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32799]: starting. 2024-09-15T04:03:27.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32799]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32763 2024-09-15T04:03:27.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32799]: created object 0... 2024-09-15T04:03:27.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32799]: created object 25... 2024-09-15T04:03:27.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32799]: created object 49... 2024-09-15T04:03:27.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32799]: finishing. 2024-09-15T04:03:27.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32799]: shutting down. 2024-09-15T04:03:27.960 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:27.961 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32801]: starting. 2024-09-15T04:03:27.961 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32801]: listing objects. 2024-09-15T04:03:27.961 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32801]: listed object 0... 2024-09-15T04:03:27.961 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32801]: listed object 25... 2024-09-15T04:03:27.961 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32801]: saw 50 objects 2024-09-15T04:03:27.961 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32801]: shutting down. 2024-09-15T04:03:28.827 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:28.827 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32800]: starting. 2024-09-15T04:03:28.827 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32800]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32763 2024-09-15T04:03:28.827 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32800]: shutting down. 2024-09-15T04:03:28.828 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:28.828 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:28.828 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:03:28.830 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:03:30.998 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32868]: starting. 2024-09-15T04:03:30.998 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32868]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32867 2024-09-15T04:03:30.998 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32868]: created object 0... 2024-09-15T04:03:30.998 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32868]: created object 25... 2024-09-15T04:03:30.998 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32868]: created object 49... 2024-09-15T04:03:30.998 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32868]: finishing. 2024-09-15T04:03:30.999 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32868]: shutting down. 2024-09-15T04:03:31.795 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32869]: starting. 2024-09-15T04:03:31.795 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32869]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32867 2024-09-15T04:03:31.795 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32869]: shutting down. 2024-09-15T04:03:33.908 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:33.908 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32902]: starting. 2024-09-15T04:03:33.908 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32902]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32867 2024-09-15T04:03:33.908 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32902]: created object 0... 2024-09-15T04:03:33.908 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32902]: created object 25... 2024-09-15T04:03:33.908 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32902]: created object 49... 2024-09-15T04:03:33.908 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32902]: finishing. 2024-09-15T04:03:33.908 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[32902]: shutting down. 2024-09-15T04:03:33.951 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:33.952 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32904]: starting. 2024-09-15T04:03:33.952 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32904]: listing objects. 2024-09-15T04:03:33.952 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32904]: listed object 0... 2024-09-15T04:03:33.952 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32904]: listed object 25... 2024-09-15T04:03:33.952 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32904]: saw 50 objects 2024-09-15T04:03:33.952 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[32904]: shutting down. 2024-09-15T04:03:34.805 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:34.805 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32903]: starting. 2024-09-15T04:03:34.805 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32903]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32867 2024-09-15T04:03:34.805 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[32903]: shutting down. 2024-09-15T04:03:34.806 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:34.806 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:34.806 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:03:34.808 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:03:37.697 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32971]: starting. 2024-09-15T04:03:37.697 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32971]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32970 2024-09-15T04:03:37.697 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32971]: created object 0... 2024-09-15T04:03:37.698 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32971]: created object 25... 2024-09-15T04:03:37.698 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32971]: created object 49... 2024-09-15T04:03:37.698 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32971]: finishing. 2024-09-15T04:03:37.698 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[32971]: shutting down. 2024-09-15T04:03:39.266 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32972]: starting. 2024-09-15T04:03:39.266 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32972]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32970 2024-09-15T04:03:39.266 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[32972]: shutting down. 2024-09-15T04:03:41.539 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:41.539 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33005]: starting. 2024-09-15T04:03:41.540 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33005]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-32970 2024-09-15T04:03:41.540 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33005]: created object 0... 2024-09-15T04:03:41.540 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33005]: created object 25... 2024-09-15T04:03:41.540 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33005]: created object 49... 2024-09-15T04:03:41.540 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33005]: finishing. 2024-09-15T04:03:41.540 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33005]: shutting down. 2024-09-15T04:03:41.579 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:41.579 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33007]: starting. 2024-09-15T04:03:41.579 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33007]: listing objects. 2024-09-15T04:03:41.579 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33007]: listed object 0... 2024-09-15T04:03:41.579 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33007]: listed object 25... 2024-09-15T04:03:41.580 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33007]: saw 50 objects 2024-09-15T04:03:41.580 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33007]: shutting down. 2024-09-15T04:03:42.278 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:42.278 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33006]: starting. 2024-09-15T04:03:42.278 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33006]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-32970 2024-09-15T04:03:42.278 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33006]: shutting down. 2024-09-15T04:03:42.279 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:42.279 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:42.279 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:03:42.282 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:03:44.369 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33074]: starting. 2024-09-15T04:03:44.369 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33074]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33073 2024-09-15T04:03:44.370 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33074]: created object 0... 2024-09-15T04:03:44.370 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33074]: created object 25... 2024-09-15T04:03:44.370 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33074]: created object 49... 2024-09-15T04:03:44.370 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33074]: finishing. 2024-09-15T04:03:44.370 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33074]: shutting down. 2024-09-15T04:03:45.299 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33075]: starting. 2024-09-15T04:03:45.300 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33075]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33073 2024-09-15T04:03:45.300 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33075]: shutting down. 2024-09-15T04:03:46.876 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:46.876 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33108]: starting. 2024-09-15T04:03:46.876 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33108]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33073 2024-09-15T04:03:46.877 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33108]: created object 0... 2024-09-15T04:03:46.877 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33108]: created object 25... 2024-09-15T04:03:46.877 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33108]: created object 49... 2024-09-15T04:03:46.877 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33108]: finishing. 2024-09-15T04:03:46.877 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33108]: shutting down. 2024-09-15T04:03:46.915 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:46.915 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33110]: starting. 2024-09-15T04:03:46.915 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33110]: listing objects. 2024-09-15T04:03:46.915 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33110]: listed object 0... 2024-09-15T04:03:46.915 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33110]: listed object 25... 2024-09-15T04:03:46.915 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33110]: saw 50 objects 2024-09-15T04:03:46.915 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33110]: shutting down. 2024-09-15T04:03:47.800 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:47.800 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33109]: starting. 2024-09-15T04:03:47.800 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33109]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33073 2024-09-15T04:03:47.801 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33109]: shutting down. 2024-09-15T04:03:47.801 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:47.801 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:47.801 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:03:47.802 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:03:50.458 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33177]: starting. 2024-09-15T04:03:50.458 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33177]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33176 2024-09-15T04:03:50.458 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33177]: created object 0... 2024-09-15T04:03:50.458 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33177]: created object 25... 2024-09-15T04:03:50.458 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33177]: created object 49... 2024-09-15T04:03:50.458 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33177]: finishing. 2024-09-15T04:03:50.458 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33177]: shutting down. 2024-09-15T04:03:51.368 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33178]: starting. 2024-09-15T04:03:51.368 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33178]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33176 2024-09-15T04:03:51.368 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33178]: shutting down. 2024-09-15T04:03:53.501 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:53.502 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33211]: starting. 2024-09-15T04:03:53.502 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33211]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33176 2024-09-15T04:03:53.502 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33211]: created object 0... 2024-09-15T04:03:53.502 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33211]: created object 25... 2024-09-15T04:03:53.502 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33211]: created object 49... 2024-09-15T04:03:53.502 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33211]: finishing. 2024-09-15T04:03:53.502 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33211]: shutting down. 2024-09-15T04:03:53.542 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:53.542 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33213]: starting. 2024-09-15T04:03:53.542 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33213]: listing objects. 2024-09-15T04:03:53.542 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33213]: listed object 0... 2024-09-15T04:03:53.544 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33213]: listed object 25... 2024-09-15T04:03:53.544 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33213]: saw 50 objects 2024-09-15T04:03:53.544 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33213]: shutting down. 2024-09-15T04:03:54.414 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:54.414 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33212]: starting. 2024-09-15T04:03:54.414 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33212]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33176 2024-09-15T04:03:54.414 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33212]: shutting down. 2024-09-15T04:03:54.415 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:54.415 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:54.415 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:03:54.417 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:03:56.507 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33280]: starting. 2024-09-15T04:03:56.507 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33280]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33279 2024-09-15T04:03:56.507 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33280]: created object 0... 2024-09-15T04:03:56.507 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33280]: created object 25... 2024-09-15T04:03:56.508 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33280]: created object 49... 2024-09-15T04:03:56.508 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33280]: finishing. 2024-09-15T04:03:56.508 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33280]: shutting down. 2024-09-15T04:03:57.801 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33281]: starting. 2024-09-15T04:03:57.802 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33281]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33279 2024-09-15T04:03:57.802 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33281]: shutting down. 2024-09-15T04:03:59.901 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:59.901 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33314]: starting. 2024-09-15T04:03:59.901 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33314]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33279 2024-09-15T04:03:59.901 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33314]: created object 0... 2024-09-15T04:03:59.901 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33314]: created object 25... 2024-09-15T04:03:59.901 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33314]: created object 49... 2024-09-15T04:03:59.901 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33314]: finishing. 2024-09-15T04:03:59.901 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33314]: shutting down. 2024-09-15T04:03:59.939 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:03:59.939 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33316]: starting. 2024-09-15T04:03:59.939 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33316]: listing objects. 2024-09-15T04:03:59.939 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33316]: listed object 0... 2024-09-15T04:03:59.939 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33316]: listed object 25... 2024-09-15T04:03:59.939 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33316]: saw 50 objects 2024-09-15T04:03:59.940 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33316]: shutting down. 2024-09-15T04:04:00.806 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:00.806 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33315]: starting. 2024-09-15T04:04:00.806 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33315]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33279 2024-09-15T04:04:00.806 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33315]: shutting down. 2024-09-15T04:04:00.807 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:00.807 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:00.807 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:04:00.810 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:04:02.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33383]: starting. 2024-09-15T04:04:02.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33383]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33382 2024-09-15T04:04:02.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33383]: created object 0... 2024-09-15T04:04:02.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33383]: created object 25... 2024-09-15T04:04:02.922 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33383]: created object 49... 2024-09-15T04:04:02.923 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33383]: finishing. 2024-09-15T04:04:02.923 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33383]: shutting down. 2024-09-15T04:04:03.816 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33384]: starting. 2024-09-15T04:04:03.816 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33384]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33382 2024-09-15T04:04:03.816 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33384]: shutting down. 2024-09-15T04:04:05.977 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:05.977 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33417]: starting. 2024-09-15T04:04:05.977 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33417]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33382 2024-09-15T04:04:05.977 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33417]: created object 0... 2024-09-15T04:04:05.977 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33417]: created object 25... 2024-09-15T04:04:05.977 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33417]: created object 49... 2024-09-15T04:04:05.978 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33417]: finishing. 2024-09-15T04:04:05.978 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33417]: shutting down. 2024-09-15T04:04:06.012 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:06.013 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33419]: starting. 2024-09-15T04:04:06.013 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33419]: listing objects. 2024-09-15T04:04:06.013 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33419]: listed object 0... 2024-09-15T04:04:06.013 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33419]: listed object 25... 2024-09-15T04:04:06.013 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33419]: saw 50 objects 2024-09-15T04:04:06.013 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33419]: shutting down. 2024-09-15T04:04:06.803 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:06.803 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33418]: starting. 2024-09-15T04:04:06.803 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33418]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33382 2024-09-15T04:04:06.803 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33418]: shutting down. 2024-09-15T04:04:06.803 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:06.804 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:06.804 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:04:06.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:04:08.947 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33486]: starting. 2024-09-15T04:04:08.948 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33486]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33485 2024-09-15T04:04:08.948 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33486]: created object 0... 2024-09-15T04:04:08.948 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33486]: created object 25... 2024-09-15T04:04:08.948 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33486]: created object 49... 2024-09-15T04:04:08.948 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33486]: finishing. 2024-09-15T04:04:08.948 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33486]: shutting down. 2024-09-15T04:04:09.840 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33487]: starting. 2024-09-15T04:04:09.840 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33487]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33485 2024-09-15T04:04:09.841 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33487]: shutting down. 2024-09-15T04:04:11.942 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:11.942 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33520]: starting. 2024-09-15T04:04:11.942 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33520]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33485 2024-09-15T04:04:11.942 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33520]: created object 0... 2024-09-15T04:04:11.942 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33520]: created object 25... 2024-09-15T04:04:11.942 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33520]: created object 49... 2024-09-15T04:04:11.943 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33520]: finishing. 2024-09-15T04:04:11.943 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33520]: shutting down. 2024-09-15T04:04:11.981 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:11.981 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33522]: starting. 2024-09-15T04:04:11.981 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33522]: listing objects. 2024-09-15T04:04:11.981 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33522]: listed object 0... 2024-09-15T04:04:11.981 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33522]: listed object 25... 2024-09-15T04:04:11.981 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33522]: saw 50 objects 2024-09-15T04:04:11.981 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33522]: shutting down. 2024-09-15T04:04:12.851 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:12.851 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33521]: starting. 2024-09-15T04:04:12.851 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33521]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33485 2024-09-15T04:04:12.851 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33521]: shutting down. 2024-09-15T04:04:12.852 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:12.853 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:12.853 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:04:12.855 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:04:14.947 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33589]: starting. 2024-09-15T04:04:14.947 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33589]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33588 2024-09-15T04:04:14.947 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33589]: created object 0... 2024-09-15T04:04:14.947 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33589]: created object 25... 2024-09-15T04:04:14.947 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33589]: created object 49... 2024-09-15T04:04:14.947 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33589]: finishing. 2024-09-15T04:04:14.947 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33589]: shutting down. 2024-09-15T04:04:15.868 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33590]: starting. 2024-09-15T04:04:15.869 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33590]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33588 2024-09-15T04:04:15.869 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33590]: shutting down. 2024-09-15T04:04:18.244 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:18.244 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33623]: starting. 2024-09-15T04:04:18.244 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33623]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33588 2024-09-15T04:04:18.244 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33623]: created object 0... 2024-09-15T04:04:18.244 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33623]: created object 25... 2024-09-15T04:04:18.244 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33623]: created object 49... 2024-09-15T04:04:18.244 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33623]: finishing. 2024-09-15T04:04:18.244 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33623]: shutting down. 2024-09-15T04:04:18.289 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:18.289 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33625]: starting. 2024-09-15T04:04:18.289 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33625]: listing objects. 2024-09-15T04:04:18.289 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33625]: listed object 0... 2024-09-15T04:04:18.289 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33625]: listed object 25... 2024-09-15T04:04:18.289 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33625]: saw 50 objects 2024-09-15T04:04:18.289 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33625]: shutting down. 2024-09-15T04:04:19.155 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:19.155 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33624]: starting. 2024-09-15T04:04:19.155 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33624]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33588 2024-09-15T04:04:19.155 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33624]: shutting down. 2024-09-15T04:04:19.156 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:19.156 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:19.157 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:04:19.159 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:04:21.251 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33692]: starting. 2024-09-15T04:04:21.252 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33692]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33691 2024-09-15T04:04:21.252 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33692]: created object 0... 2024-09-15T04:04:21.252 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33692]: created object 25... 2024-09-15T04:04:21.252 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33692]: created object 49... 2024-09-15T04:04:21.252 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33692]: finishing. 2024-09-15T04:04:21.252 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33692]: shutting down. 2024-09-15T04:04:22.177 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33693]: starting. 2024-09-15T04:04:22.177 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33693]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33691 2024-09-15T04:04:22.177 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33693]: shutting down. 2024-09-15T04:04:24.304 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:24.304 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33726]: starting. 2024-09-15T04:04:24.304 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33726]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33691 2024-09-15T04:04:24.304 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33726]: created object 0... 2024-09-15T04:04:24.304 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33726]: created object 25... 2024-09-15T04:04:24.304 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33726]: created object 49... 2024-09-15T04:04:24.305 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33726]: finishing. 2024-09-15T04:04:24.305 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33726]: shutting down. 2024-09-15T04:04:24.342 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:24.342 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33728]: starting. 2024-09-15T04:04:24.342 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33728]: listing objects. 2024-09-15T04:04:24.342 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33728]: listed object 0... 2024-09-15T04:04:24.342 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33728]: listed object 25... 2024-09-15T04:04:24.343 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33728]: saw 50 objects 2024-09-15T04:04:24.343 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33728]: shutting down. 2024-09-15T04:04:25.195 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:25.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33727]: starting. 2024-09-15T04:04:25.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33727]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33691 2024-09-15T04:04:25.196 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33727]: shutting down. 2024-09-15T04:04:25.196 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:25.196 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:25.197 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:04:25.199 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:04:26.927 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33795]: starting. 2024-09-15T04:04:26.928 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33795]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33794 2024-09-15T04:04:26.928 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33795]: created object 0... 2024-09-15T04:04:26.928 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33795]: created object 25... 2024-09-15T04:04:26.928 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33795]: created object 49... 2024-09-15T04:04:26.928 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33795]: finishing. 2024-09-15T04:04:26.928 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33795]: shutting down. 2024-09-15T04:04:27.809 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33796]: starting. 2024-09-15T04:04:27.809 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33796]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33794 2024-09-15T04:04:27.809 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33796]: shutting down. 2024-09-15T04:04:29.925 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:29.925 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33829]: starting. 2024-09-15T04:04:29.925 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33829]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33794 2024-09-15T04:04:29.925 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33829]: created object 0... 2024-09-15T04:04:29.925 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33829]: created object 25... 2024-09-15T04:04:29.925 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33829]: created object 49... 2024-09-15T04:04:29.925 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33829]: finishing. 2024-09-15T04:04:29.926 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33829]: shutting down. 2024-09-15T04:04:29.965 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:29.965 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33831]: starting. 2024-09-15T04:04:29.965 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33831]: listing objects. 2024-09-15T04:04:29.965 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33831]: listed object 0... 2024-09-15T04:04:29.965 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33831]: listed object 25... 2024-09-15T04:04:29.965 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33831]: saw 50 objects 2024-09-15T04:04:29.965 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33831]: shutting down. 2024-09-15T04:04:30.823 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:30.823 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33830]: starting. 2024-09-15T04:04:30.824 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33830]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33794 2024-09-15T04:04:30.824 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33830]: shutting down. 2024-09-15T04:04:30.824 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:30.825 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:30.825 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:04:30.827 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:04:33.126 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33899]: starting. 2024-09-15T04:04:33.127 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33899]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33898 2024-09-15T04:04:33.127 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33899]: created object 0... 2024-09-15T04:04:33.127 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33899]: created object 25... 2024-09-15T04:04:33.127 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33899]: created object 49... 2024-09-15T04:04:33.127 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33899]: finishing. 2024-09-15T04:04:33.127 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[33899]: shutting down. 2024-09-15T04:04:33.837 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33900]: starting. 2024-09-15T04:04:33.837 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33900]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33898 2024-09-15T04:04:33.837 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[33900]: shutting down. 2024-09-15T04:04:36.148 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:36.148 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33933]: starting. 2024-09-15T04:04:36.149 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33933]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-33898 2024-09-15T04:04:36.149 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33933]: created object 0... 2024-09-15T04:04:36.149 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33933]: created object 25... 2024-09-15T04:04:36.149 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33933]: created object 49... 2024-09-15T04:04:36.149 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33933]: finishing. 2024-09-15T04:04:36.149 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[33933]: shutting down. 2024-09-15T04:04:36.189 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:36.189 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33935]: starting. 2024-09-15T04:04:36.189 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33935]: listing objects. 2024-09-15T04:04:36.189 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33935]: listed object 0... 2024-09-15T04:04:36.189 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33935]: listed object 25... 2024-09-15T04:04:36.189 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33935]: saw 50 objects 2024-09-15T04:04:36.189 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[33935]: shutting down. 2024-09-15T04:04:36.809 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:36.810 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33934]: starting. 2024-09-15T04:04:36.810 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33934]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-33898 2024-09-15T04:04:36.810 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[33934]: shutting down. 2024-09-15T04:04:36.811 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:36.811 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:36.811 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:04:36.813 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:04:38.949 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[34002]: starting. 2024-09-15T04:04:38.949 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[34002]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-34001 2024-09-15T04:04:38.949 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[34002]: created object 0... 2024-09-15T04:04:38.949 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[34002]: created object 25... 2024-09-15T04:04:38.950 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[34002]: created object 49... 2024-09-15T04:04:38.950 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[34002]: finishing. 2024-09-15T04:04:38.950 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[34002]: shutting down. 2024-09-15T04:04:39.866 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[34003]: starting. 2024-09-15T04:04:39.866 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[34003]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-34001 2024-09-15T04:04:39.866 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[34003]: shutting down. 2024-09-15T04:04:42.284 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:42.284 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[34036]: starting. 2024-09-15T04:04:42.284 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[34036]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-34001 2024-09-15T04:04:42.284 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[34036]: created object 0... 2024-09-15T04:04:42.284 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[34036]: created object 25... 2024-09-15T04:04:42.284 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[34036]: created object 49... 2024-09-15T04:04:42.284 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[34036]: finishing. 2024-09-15T04:04:42.284 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[34036]: shutting down. 2024-09-15T04:04:42.327 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:42.327 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[34038]: starting. 2024-09-15T04:04:42.327 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[34038]: listing objects. 2024-09-15T04:04:42.327 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[34038]: listed object 0... 2024-09-15T04:04:42.327 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[34038]: listed object 25... 2024-09-15T04:04:42.327 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[34038]: saw 50 objects 2024-09-15T04:04:42.327 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[34038]: shutting down. 2024-09-15T04:04:42.890 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:42.890 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[34037]: starting. 2024-09-15T04:04:42.890 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[34037]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-34001 2024-09-15T04:04:42.891 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[34037]: shutting down. 2024-09-15T04:04:42.891 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:42.891 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:42.891 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:04:42.893 DEBUG:teuthology.orchestra.run.smithi177:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-09-15T04:04:44.994 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[34105]: starting. 2024-09-15T04:04:44.995 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[34105]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-34104 2024-09-15T04:04:44.995 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[34105]: created object 0... 2024-09-15T04:04:44.995 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[34105]: created object 25... 2024-09-15T04:04:44.995 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[34105]: created object 49... 2024-09-15T04:04:44.995 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[34105]: finishing. 2024-09-15T04:04:44.995 INFO:teuthology.orchestra.run.smithi177.stdout:process_1_[34105]: shutting down. 2024-09-15T04:04:45.897 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[34106]: starting. 2024-09-15T04:04:45.898 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[34106]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-34104 2024-09-15T04:04:45.898 INFO:teuthology.orchestra.run.smithi177.stdout:process_2_[34106]: shutting down. 2024-09-15T04:04:47.895 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:47.895 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[34139]: starting. 2024-09-15T04:04:47.895 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[34139]: creating pool ceph_test_rados_delete_pools_parallel.smithi177-34104 2024-09-15T04:04:47.896 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[34139]: created object 0... 2024-09-15T04:04:47.896 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[34139]: created object 25... 2024-09-15T04:04:47.896 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[34139]: created object 49... 2024-09-15T04:04:47.896 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[34139]: finishing. 2024-09-15T04:04:47.896 INFO:teuthology.orchestra.run.smithi177.stdout:process_3_[34139]: shutting down. 2024-09-15T04:04:47.934 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:04:47.935 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[34141]: starting. 2024-09-15T04:04:47.935 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[34141]: listing objects. 2024-09-15T04:04:47.935 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[34141]: listed object 0... 2024-09-15T04:04:47.935 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[34141]: listed object 25... 2024-09-15T04:04:47.935 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[34141]: saw 50 objects 2024-09-15T04:04:47.935 INFO:teuthology.orchestra.run.smithi177.stdout:process_5_[34141]: shutting down. 2024-09-15T04:04:48.153 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T04:04:48.154 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T04:04:48.154 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T04:04:48.154 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T04:04:48.157 INFO:tasks.ceph.mon.c:Started 2024-09-15T04:04:48.157 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T04:04:48.157 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T04:04:48.158 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T04:05:03.159 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T04:05:03.160 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T04:05:03.160 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T04:05:03.160 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:05:03.181 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:05:03.181 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[34140]: starting. 2024-09-15T04:05:03.181 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[34140]: deleting pool ceph_test_rados_delete_pools_parallel.smithi177-34104 2024-09-15T04:05:03.181 INFO:teuthology.orchestra.run.smithi177.stdout:process_4_[34140]: shutting down. 2024-09-15T04:05:03.182 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:05:03.182 INFO:teuthology.orchestra.run.smithi177.stdout:******************************* 2024-09-15T04:05:03.182 INFO:teuthology.orchestra.run.smithi177.stdout:******* SUCCESS ********** 2024-09-15T04:05:03.184 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-15T04:05:03.194 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-15T04:05:03.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.263+0000 7f4a5f79d640 10 monclient: get_monmap_and_config 2024-09-15T04:05:03.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.263+0000 7f4a5f79d640 10 monclient: build_initial_monmap 2024-09-15T04:05:03.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.263+0000 7f4a5f79d640 10 monclient: monmap: 2024-09-15T04:05:03.264 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:03.264 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.265 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:03.264655+0000 2024-09-15T04:05:03.265 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:03.264655+0000 2024-09-15T04:05:03.265 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:03.265 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.265 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:03.265 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:03.265 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:03.265 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:03.266 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:03.266 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 1 Processor -- start 2024-09-15T04:05:03.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 1 -- start start 2024-09-15T04:05:03.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 10 monclient: init 2024-09-15T04:05:03.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:03.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-15T04:05:03.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4a58131630 0x7f4a58131a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 10 monclient(hunting): picked mon.noname-c con 0x7f4a58131630 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:03.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4a58131f40 0x7f4a5806e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 10 monclient(hunting): picked mon.noname-a con 0x7f4a58131f40 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:03.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a5806eae0 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:03.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5809f320 con 0x7f4a5806eae0 2024-09-15T04:05:03.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a58132340 con 0x7f4a58131f40 2024-09-15T04:05:03.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a58068390 con 0x7f4a58131630 2024-09-15T04:05:03.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:03.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.264+0000 7f4a5f79d640 10 monclient(hunting): authenticate will time out at 1126.908986s 2024-09-15T04:05:03.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4a58131f40 0x7f4a5806e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4a58131f40 0x7f4a5806e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42678/0 (socket says 172.21.15.110:42678) 2024-09-15T04:05:03.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 1 -- 172.21.15.110:0/3830129103 learned_addr learned my addr 172.21.15.110:0/3830129103 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:03.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 10 monclient(hunting): get_auth_request con 0x7f4a58131f40 auth_method 0 2024-09-15T04:05:03.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5d539640 1 --2- 172.21.15.110:0/3830129103 >> v2:172.21.15.177:3300/0 conn(0x7f4a58131630 0x7f4a58131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5d539640 10 monclient(hunting): get_auth_request con 0x7f4a58131630 auth_method 0 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5d539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5d539640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5d539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 10 monclient(hunting): handle_auth_done global_id 55152 payload 274 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 1 -- 172.21.15.110:0/3830129103 >> v2:172.21.15.177:3300/0 conn(0x7f4a58131630 msgr2=0x7f4a58131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 1 --2- 172.21.15.110:0/3830129103 >> v2:172.21.15.177:3300/0 conn(0x7f4a58131630 0x7f4a58131a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 1 -- 172.21.15.110:0/3830129103 >> v1:172.21.15.177:6789/0 conn(0x7f4a5806eae0 legacy=0x7f4a5809d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:03.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 10 monclient: _finish_hunting 0 2024-09-15T04:05:03.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 1 monclient: found mon.noname-a 2024-09-15T04:05:03.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:03.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:03.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.265+0000 7f4a5cd38640 1 -- 172.21.15.110:0/3830129103 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a58069570 con 0x7f4a58131f40 2024-09-15T04:05:03.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a5cd38640 1 -- 172.21.15.110:0/3830129103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a58131f40 msgr2=0x7f4a5806e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a5cd38640 1 --2- 172.21.15.110:0/3830129103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a58131f40 0x7f4a5806e5a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4a5806bed0 tx=0x7f4a480091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc6684b2eb936e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a4ffff640 1 -- 172.21.15.110:0/3830129103 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48032440 con 0x7f4a58131f40 2024-09-15T04:05:03.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a4ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:03.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a4ffff640 10 monclient: dump: 2024-09-15T04:05:03.277 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.277 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.277 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.278 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.278 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.278 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.278 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.278 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.278 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.278 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a4ffff640 10 monclient: _finish_auth 0 2024-09-15T04:05:03.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:03.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a4ffff640 1 -- 172.21.15.110:0/3830129103 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a480325a0 con 0x7f4a58131f40 2024-09-15T04:05:03.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a5f79d640 5 monclient: authenticate success, global_id 55152 2024-09-15T04:05:03.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a4ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:03.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a5f79d640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:03.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a4ffff640 1 -- 172.21.15.110:0/3830129103 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48032940 con 0x7f4a58131f40 2024-09-15T04:05:03.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a5f79d640 4 set_mon_vals no callback set 2024-09-15T04:05:03.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a4ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:03.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a4ffff640 10 monclient: dump: 2024-09-15T04:05:03.279 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.279 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.279 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.279 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.279 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.279 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.279 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.280 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.280 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.280 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a5f79d640 10 monclient: shutdown 2024-09-15T04:05:03.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a5f79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:03.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a5f79d640 1 -- 172.21.15.110:0/3830129103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a58131f40 msgr2=0x7f4a5806e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a5f79d640 1 --2- 172.21.15.110:0/3830129103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a58131f40 0x7f4a5806e5a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4a5806bed0 tx=0x7f4a480091d0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a5f79d640 1 -- 172.21.15.110:0/3830129103 shutdown_connections 2024-09-15T04:05:03.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a5f79d640 1 --2- 172.21.15.110:0/3830129103 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a58131f40 0x7f4a5806e5a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a5f79d640 1 --2- 172.21.15.110:0/3830129103 >> v2:172.21.15.177:3300/0 conn(0x7f4a58131630 0x7f4a58131a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.268+0000 7f4a5f79d640 1 -- 172.21.15.110:0/3830129103 >> 172.21.15.110:0/3830129103 conn(0x7f4a580639f0 msgr2=0x7f4a580a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:03.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.269+0000 7f4a5f79d640 1 -- 172.21.15.110:0/3830129103 shutdown_connections 2024-09-15T04:05:03.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.269+0000 7f4a5f79d640 1 -- 172.21.15.110:0/3830129103 wait complete. 2024-09-15T04:05:03.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.269+0000 7f4a5f79d640 10 monclient: build_initial_monmap 2024-09-15T04:05:03.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.269+0000 7f4a5f79d640 10 monclient: monmap: 2024-09-15T04:05:03.281 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:03.281 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:03.281 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:03.281 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:03.281 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:03.281 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.281 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:03.281 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:03.281 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:03.281 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.269+0000 7f4a5f79d640 1 Processor -- start 2024-09-15T04:05:03.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.269+0000 7f4a5f79d640 1 -- start start 2024-09-15T04:05:03.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.269+0000 7f4a5f79d640 10 monclient: init 2024-09-15T04:05:03.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:03.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T04:05:03.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a5806eae0 0x7f4a5812eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 10 monclient(hunting): picked mon.noname-b con 0x7f4a5806eae0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:03.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a58131630 0x7f4a5811b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 10 monclient(hunting): picked mon.noname-a con 0x7f4a58131630 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:03.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a58131f40 0x7f4a5811bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 10 monclient(hunting): picked mon.noname-c con 0x7f4a58131f40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:03.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5811f490 con 0x7f4a58131630 2024-09-15T04:05:03.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580688c0 con 0x7f4a5806eae0 2024-09-15T04:05:03.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a580a2610 con 0x7f4a58131f40 2024-09-15T04:05:03.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:03.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5f79d640 10 monclient(hunting): authenticate will time out at 1126.914534s 2024-09-15T04:05:03.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a58131630 0x7f4a5811b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a58131630 0x7f4a5811b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42688/0 (socket says 172.21.15.110:42688) 2024-09-15T04:05:03.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 1 -- 172.21.15.110:0/1597841423 learned_addr learned my addr 172.21.15.110:0/1597841423 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:03.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 10 monclient(hunting): get_auth_request con 0x7f4a58131630 auth_method 0 2024-09-15T04:05:03.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5dd3a640 1 --2- 172.21.15.110:0/1597841423 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a58131f40 0x7f4a5811bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5dd3a640 10 monclient(hunting): get_auth_request con 0x7f4a58131f40 auth_method 0 2024-09-15T04:05:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5dd3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5dd3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5dd3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5d539640 1 --2- 172.21.15.110:0/1597841423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a5806eae0 0x7f4a5812eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5dd3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5dd3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5dd3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5d539640 10 monclient(hunting): get_auth_request con 0x7f4a5806eae0 auth_method 0 2024-09-15T04:05:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5d539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5d539640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5d539640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5d539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5d539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5d539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 10 monclient(hunting): handle_auth_done global_id 55155 payload 824 2024-09-15T04:05:03.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 1 -- 172.21.15.110:0/1597841423 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a58131f40 msgr2=0x7f4a5811bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 1 --2- 172.21.15.110:0/1597841423 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a58131f40 0x7f4a5811bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 1 -- 172.21.15.110:0/1597841423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a5806eae0 msgr2=0x7f4a5812eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 1 --2- 172.21.15.110:0/1597841423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a5806eae0 0x7f4a5812eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 10 monclient: _finish_hunting 0 2024-09-15T04:05:03.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 1 monclient: found mon.noname-a 2024-09-15T04:05:03.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:03.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:03.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.270+0000 7f4a5cd38640 1 -- 172.21.15.110:0/1597841423 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a48009450 con 0x7f4a58131630 2024-09-15T04:05:03.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a5cd38640 1 -- 172.21.15.110:0/1597841423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a58131630 msgr2=0x7f4a5811b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a5cd38640 1 --2- 172.21.15.110:0/1597841423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a58131630 0x7f4a5811b7c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4a48009940 tx=0x7f4a48032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a4d7fa640 1 -- 172.21.15.110:0/1597841423 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48031a40 con 0x7f4a58131630 2024-09-15T04:05:03.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a4d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:03.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a4d7fa640 10 monclient: dump: 2024-09-15T04:05:03.286 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.286 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.286 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.286 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.286 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.286 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.286 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.286 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.286 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a4d7fa640 10 monclient: _finish_auth 0 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a4d7fa640 1 -- 172.21.15.110:0/1597841423 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a48031be0 con 0x7f4a58131630 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a5f79d640 5 monclient: authenticate success, global_id 55155 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a4d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a4d7fa640 1 -- 172.21.15.110:0/1597841423 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4803d440 con 0x7f4a58131630 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a4d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a4d7fa640 10 monclient: dump: 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.287 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.288 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.288 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.288 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.288 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.288 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a5f79d640 10 monclient: _renew_subs 2024-09-15T04:05:03.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a5f79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:03.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a5f79d640 1 -- 172.21.15.110:0/1597841423 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5811fb20 con 0x7f4a58131630 2024-09-15T04:05:03.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a4f7fe640 4 set_mon_vals no callback set 2024-09-15T04:05:03.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a5f79d640 10 monclient: _renew_subs 2024-09-15T04:05:03.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a5f79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:03.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.273+0000 7f4a5f79d640 1 -- 172.21.15.110:0/1597841423 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5811dcc0 con 0x7f4a58131630 2024-09-15T04:05:03.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.274+0000 7f4a4d7fa640 1 -- 172.21.15.110:0/1597841423 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f4a48039040 con 0x7f4a58131630 2024-09-15T04:05:03.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.274+0000 7f4a5f79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:03.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.274+0000 7f4a5f79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:03.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.274+0000 7f4a5f79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:03.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.274+0000 7f4a5f79d640 1 -- 172.21.15.110:0/1597841423 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a58070950 con 0x7f4a58131630 2024-09-15T04:05:03.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.274+0000 7f4a4d7fa640 1 --2- 172.21.15.110:0/1597841423 >> v2:172.21.15.177:6812/28201 conn(0x7f4a3c038700 0x7f4a3c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.274+0000 7f4a4d7fa640 1 -- 172.21.15.110:0/1597841423 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4a4806b410 con 0x7f4a58131630 2024-09-15T04:05:03.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.274+0000 7f4a5d539640 1 --2- 172.21.15.110:0/1597841423 >> v2:172.21.15.177:6812/28201 conn(0x7f4a3c038700 0x7f4a3c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.274+0000 7f4a5d539640 10 monclient: get_auth_request con 0x7f4a3c038700 auth_method 0 2024-09-15T04:05:03.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.277+0000 7f4a5d539640 1 --2- 172.21.15.110:0/1597841423 >> v2:172.21.15.177:6812/28201 conn(0x7f4a3c038700 0x7f4a3c03abb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4a54007ba0 tx=0x7f4a54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.278+0000 7f4a4d7fa640 1 -- 172.21.15.110:0/1597841423 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4a4803d7f0 con 0x7f4a58131630 2024-09-15T04:05:03.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.278+0000 7f4a4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:03.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.278+0000 7f4a4d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:03.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.491+0000 7f4a5f79d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:05:03.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.491+0000 7f4a5f79d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:05:03.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.491+0000 7f4a5f79d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:03.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.491+0000 7f4a5f79d640 1 -- 172.21.15.110:0/1597841423 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4a5811d3a0 con 0x7f4a58131630 2024-09-15T04:05:03.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.492+0000 7f4a4d7fa640 1 -- 172.21.15.110:0/1597841423 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f4a48037080 con 0x7f4a58131630 2024-09-15T04:05:03.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.492+0000 7f4a4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:05:03.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.492+0000 7f4a4d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:05:03.493 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:05:03.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.494+0000 7f4a5f79d640 1 -- 172.21.15.110:0/1597841423 >> v2:172.21.15.177:6812/28201 conn(0x7f4a3c038700 msgr2=0x7f4a3c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.494+0000 7f4a5f79d640 1 --2- 172.21.15.110:0/1597841423 >> v2:172.21.15.177:6812/28201 conn(0x7f4a3c038700 0x7f4a3c03abb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4a54007ba0 tx=0x7f4a54008040 comp rx=0 tx=0).stop 2024-09-15T04:05:03.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.494+0000 7f4a5f79d640 10 monclient: shutdown 2024-09-15T04:05:03.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.494+0000 7f4a5f79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:03.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.494+0000 7f4a5f79d640 1 -- 172.21.15.110:0/1597841423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a58131630 msgr2=0x7f4a5811b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.494+0000 7f4a5f79d640 1 --2- 172.21.15.110:0/1597841423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a58131630 0x7f4a5811b7c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f4a48009940 tx=0x7f4a48032000 comp rx=0 tx=0).stop 2024-09-15T04:05:03.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.494+0000 7f4a5f79d640 1 -- 172.21.15.110:0/1597841423 shutdown_connections 2024-09-15T04:05:03.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.494+0000 7f4a5f79d640 1 --2- 172.21.15.110:0/1597841423 >> v2:172.21.15.177:6812/28201 conn(0x7f4a3c038700 0x7f4a3c03abb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.494+0000 7f4a5f79d640 1 --2- 172.21.15.110:0/1597841423 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a58131f40 0x7f4a5811bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.494+0000 7f4a5f79d640 1 --2- 172.21.15.110:0/1597841423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a58131630 0x7f4a5811b7c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.494+0000 7f4a5f79d640 1 --2- 172.21.15.110:0/1597841423 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a5806eae0 0x7f4a5812eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.494+0000 7f4a5f79d640 1 -- 172.21.15.110:0/1597841423 >> 172.21.15.110:0/1597841423 conn(0x7f4a580639f0 msgr2=0x7f4a5806dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:03.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.494+0000 7f4a5f79d640 1 -- 172.21.15.110:0/1597841423 shutdown_connections 2024-09-15T04:05:03.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.494+0000 7f4a5f79d640 1 -- 172.21.15.110:0/1597841423 wait complete. 2024-09-15T04:05:03.504 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T04:05:03.504 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T04:05:03.505 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T04:05:03.597 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.597+0000 7f7a182a8640 10 monclient: get_monmap_and_config 2024-09-15T04:05:03.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.597+0000 7f7a182a8640 10 monclient: build_initial_monmap 2024-09-15T04:05:03.598 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.597+0000 7f7a182a8640 10 monclient: monmap: 2024-09-15T04:05:03.598 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:03.598 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.598 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:03.598119+0000 2024-09-15T04:05:03.598 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:03.598119+0000 2024-09-15T04:05:03.598 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:03.598 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.598 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:03.598 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:03.598 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:03.598 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:03.599 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:03.599 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.597+0000 7f7a182a8640 1 Processor -- start 2024-09-15T04:05:03.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.597+0000 7f7a182a8640 1 -- start start 2024-09-15T04:05:03.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.597+0000 7f7a182a8640 10 monclient: init 2024-09-15T04:05:03.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a182a8640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:03.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a182a8640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-15T04:05:03.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a182a8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7a100a1cf0 0x7f7a1009e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.601 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a182a8640 10 monclient(hunting): picked mon.noname-c con 0x7f7a100a1cf0 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:03.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a182a8640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7a10131a80 0x7f7a1009eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a182a8640 10 monclient(hunting): picked mon.noname-a con 0x7f7a10131a80 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:03.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a182a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a1009f760 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:03.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a182a8640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a1009cb70 con 0x7f7a1009f760 2024-09-15T04:05:03.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a182a8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a182a8640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a100a20c0 con 0x7f7a10131a80 2024-09-15T04:05:03.602 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a182a8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a182a8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1009c330 con 0x7f7a100a1cf0 2024-09-15T04:05:03.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a182a8640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:03.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a182a8640 10 monclient(hunting): authenticate will time out at 1127.242512s 2024-09-15T04:05:03.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7a10131a80 0x7f7a1009eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7a10131a80 0x7f7a1009eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42690/0 (socket says 172.21.15.110:42690) 2024-09-15T04:05:03.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 1 -- 172.21.15.110:0/4049269703 learned_addr learned my addr 172.21.15.110:0/4049269703 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:03.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 10 monclient(hunting): get_auth_request con 0x7f7a10131a80 auth_method 0 2024-09-15T04:05:03.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.603 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a16044640 1 --2- 172.21.15.110:0/4049269703 >> v2:172.21.15.177:3300/0 conn(0x7f7a100a1cf0 0x7f7a1009e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a16044640 10 monclient(hunting): get_auth_request con 0x7f7a100a1cf0 auth_method 0 2024-09-15T04:05:03.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a16044640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a16044640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a16044640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 10 monclient(hunting): handle_auth_done global_id 55158 payload 274 2024-09-15T04:05:03.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 1 -- 172.21.15.110:0/4049269703 >> v2:172.21.15.177:3300/0 conn(0x7f7a100a1cf0 msgr2=0x7f7a1009e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:03.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 1 --2- 172.21.15.110:0/4049269703 >> v2:172.21.15.177:3300/0 conn(0x7f7a100a1cf0 0x7f7a1009e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 1 -- 172.21.15.110:0/4049269703 >> v1:172.21.15.177:6789/0 conn(0x7f7a1009f760 legacy=0x7f7a1009fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:03.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 10 monclient: _finish_hunting 0 2024-09-15T04:05:03.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 1 monclient: found mon.noname-a 2024-09-15T04:05:03.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:03.604 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:03.605 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.598+0000 7f7a15843640 1 -- 172.21.15.110:0/4049269703 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a10131fb0 con 0x7f7a10131a80 2024-09-15T04:05:03.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a15843640 1 -- 172.21.15.110:0/4049269703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10131a80 msgr2=0x7f7a1009eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a15843640 1 --2- 172.21.15.110:0/4049269703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10131a80 0x7f7a1009eba0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7a10063d70 tx=0x7f7a000091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=475a2611c18b4558 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a15042640 1 -- 172.21.15.110:0/4049269703 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00032440 con 0x7f7a10131a80 2024-09-15T04:05:03.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a15042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a15042640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:03.617 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a15042640 10 monclient: dump: 2024-09-15T04:05:03.617 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.617 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.617 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.617 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.617 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.617 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.617 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.617 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.618 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.618 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a15042640 10 monclient: _finish_auth 0 2024-09-15T04:05:03.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a15042640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:03.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a15042640 1 -- 172.21.15.110:0/4049269703 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a000325a0 con 0x7f7a10131a80 2024-09-15T04:05:03.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a182a8640 5 monclient: authenticate success, global_id 55158 2024-09-15T04:05:03.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a182a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:03.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a15042640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:03.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a15042640 1 -- 172.21.15.110:0/4049269703 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00032980 con 0x7f7a10131a80 2024-09-15T04:05:03.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a182a8640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:03.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a182a8640 4 set_mon_vals no callback set 2024-09-15T04:05:03.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a15042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a15042640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:03.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.601+0000 7f7a15042640 10 monclient: dump: 2024-09-15T04:05:03.619 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.619 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.620 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.620 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.620 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.620 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.620 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.620 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.620 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.620 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.602+0000 7f7a182a8640 10 monclient: shutdown 2024-09-15T04:05:03.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.602+0000 7f7a182a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:03.620 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.602+0000 7f7a182a8640 1 -- 172.21.15.110:0/4049269703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10131a80 msgr2=0x7f7a1009eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.602+0000 7f7a182a8640 1 --2- 172.21.15.110:0/4049269703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10131a80 0x7f7a1009eba0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7a10063d70 tx=0x7f7a000091d0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.602+0000 7f7a182a8640 1 -- 172.21.15.110:0/4049269703 shutdown_connections 2024-09-15T04:05:03.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.602+0000 7f7a182a8640 1 --2- 172.21.15.110:0/4049269703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10131a80 0x7f7a1009eba0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.602+0000 7f7a182a8640 1 --2- 172.21.15.110:0/4049269703 >> v2:172.21.15.177:3300/0 conn(0x7f7a100a1cf0 0x7f7a1009e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.602+0000 7f7a182a8640 1 -- 172.21.15.110:0/4049269703 >> 172.21.15.110:0/4049269703 conn(0x7f7a10062f70 msgr2=0x7f7a10063340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:03.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.602+0000 7f7a182a8640 1 -- 172.21.15.110:0/4049269703 shutdown_connections 2024-09-15T04:05:03.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.602+0000 7f7a182a8640 1 -- 172.21.15.110:0/4049269703 wait complete. 2024-09-15T04:05:03.621 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.602+0000 7f7a182a8640 10 monclient: build_initial_monmap 2024-09-15T04:05:03.622 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.602+0000 7f7a182a8640 10 monclient: monmap: 2024-09-15T04:05:03.622 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:03.622 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:03.622 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:03.622 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:03.622 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:03.622 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.622 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:03.622 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:03.622 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:03.622 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 1 Processor -- start 2024-09-15T04:05:03.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 1 -- start start 2024-09-15T04:05:03.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 10 monclient: init 2024-09-15T04:05:03.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:03.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T04:05:03.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a1009f760 0x7f7a1009a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 10 monclient(hunting): picked mon.noname-c con 0x7f7a1009f760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:03.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a100a1cf0 0x7f7a101306e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 10 monclient(hunting): picked mon.noname-b con 0x7f7a100a1cf0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:03.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10131a80 0x7f7a1011b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.623 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 10 monclient(hunting): picked mon.noname-a con 0x7f7a10131a80 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:03.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1011dae0 con 0x7f7a10131a80 2024-09-15T04:05:03.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1011ddc0 con 0x7f7a100a1cf0 2024-09-15T04:05:03.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1011e0a0 con 0x7f7a1009f760 2024-09-15T04:05:03.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:03.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a182a8640 10 monclient(hunting): authenticate will time out at 1127.247745s 2024-09-15T04:05:03.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16044640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a1009f760 0x7f7a1009a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.624 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16845640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10131a80 0x7f7a1011b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16044640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a1009f760 0x7f7a1009a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35028/0 (socket says 172.21.15.110:35028) 2024-09-15T04:05:03.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16044640 1 -- 172.21.15.110:0/3366138136 learned_addr learned my addr 172.21.15.110:0/3366138136 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:03.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16044640 10 monclient(hunting): get_auth_request con 0x7f7a1009f760 auth_method 0 2024-09-15T04:05:03.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16044640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16044640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16044640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16845640 10 monclient(hunting): get_auth_request con 0x7f7a10131a80 auth_method 0 2024-09-15T04:05:03.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16845640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.625 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16845640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16845640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a15843640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a100a1cf0 0x7f7a101306e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a15843640 10 monclient(hunting): get_auth_request con 0x7f7a100a1cf0 auth_method 0 2024-09-15T04:05:03.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a15843640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a15843640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a15843640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.626 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16044640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16044640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16044640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16845640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16845640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.603+0000 7f7a16845640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.604+0000 7f7a15843640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.604+0000 7f7a15843640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.604+0000 7f7a15843640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.604+0000 7f7a16845640 10 monclient(hunting): handle_auth_done global_id 55161 payload 824 2024-09-15T04:05:03.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.604+0000 7f7a16845640 1 -- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a1009f760 msgr2=0x7f7a1009a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.604+0000 7f7a16845640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a1009f760 0x7f7a1009a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.604+0000 7f7a16845640 1 -- 172.21.15.110:0/3366138136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a100a1cf0 msgr2=0x7f7a101306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.604+0000 7f7a16845640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7a100a1cf0 0x7f7a101306e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.627 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.604+0000 7f7a16845640 10 monclient: _finish_hunting 0 2024-09-15T04:05:03.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.604+0000 7f7a16845640 1 monclient: found mon.noname-a 2024-09-15T04:05:03.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.604+0000 7f7a16845640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:03.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.604+0000 7f7a16845640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:03.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.604+0000 7f7a16845640 1 -- 172.21.15.110:0/3366138136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a00009450 con 0x7f7a10131a80 2024-09-15T04:05:03.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.604+0000 7f7a16044640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7a1009f760 0x7f7a1009a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:03.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a16845640 1 -- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10131a80 msgr2=0x7f7a1011b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a16845640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10131a80 0x7f7a1011b3d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7a0800b810 tx=0x7f7a080092b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a067fc640 1 -- 172.21.15.110:0/3366138136 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0800c940 con 0x7f7a10131a80 2024-09-15T04:05:03.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:03.628 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a067fc640 10 monclient: dump: 2024-09-15T04:05:03.628 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.628 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.629 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.629 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.629 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.629 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.629 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.629 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.629 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.629 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.629 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a067fc640 10 monclient: _finish_auth 0 2024-09-15T04:05:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a067fc640 1 -- 172.21.15.110:0/3366138136 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0800cae0 con 0x7f7a10131a80 2024-09-15T04:05:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a182a8640 5 monclient: authenticate success, global_id 55161 2024-09-15T04:05:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a182a8640 10 monclient: _renew_subs 2024-09-15T04:05:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a182a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a182a8640 1 -- 172.21.15.110:0/3366138136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a10141200 con 0x7f7a10131a80 2024-09-15T04:05:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a067fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a067fc640 1 -- 172.21.15.110:0/3366138136 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0800cea0 con 0x7f7a10131a80 2024-09-15T04:05:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a067fc640 10 monclient: dump: 2024-09-15T04:05:03.630 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.631 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a14841640 4 set_mon_vals no callback set 2024-09-15T04:05:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a182a8640 10 monclient: _renew_subs 2024-09-15T04:05:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a182a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.606+0000 7f7a182a8640 1 -- 172.21.15.110:0/3366138136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1009b8c0 con 0x7f7a10131a80 2024-09-15T04:05:03.631 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a067fc640 1 -- 172.21.15.110:0/3366138136 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f7a08014620 con 0x7f7a10131a80 2024-09-15T04:05:03.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a182a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:03.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a182a8640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a1011c8a0 0x7f7a1011b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a182a8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a182a8640 1 -- 172.21.15.110:0/3366138136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1006f1e0 con 0x7f7a1011c8a0 2024-09-15T04:05:03.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a067fc640 1 --2- 172.21.15.110:0/3366138136 >> v2:172.21.15.177:6812/28201 conn(0x7f79f4038700 0x7f79f403abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a16044640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a1011c8a0 0x7f7a1011b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a067fc640 1 -- 172.21.15.110:0/3366138136 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7a0804af50 con 0x7f7a10131a80 2024-09-15T04:05:03.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a16044640 10 monclient: get_auth_request con 0x7f7a1011c8a0 auth_method 0 2024-09-15T04:05:03.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a16044640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a16044640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a16044640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a16044640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.632 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a16044640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a16044640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a15843640 1 --2- 172.21.15.110:0/3366138136 >> v2:172.21.15.177:6812/28201 conn(0x7f79f4038700 0x7f79f403abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a15843640 10 monclient: get_auth_request con 0x7f79f4038700 auth_method 0 2024-09-15T04:05:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a16044640 10 monclient(hunting): handle_auth_done global_id 55164 payload 824 2024-09-15T04:05:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.607+0000 7f7a16044640 1 -- 172.21.15.110:0/3366138136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a1009b440 con 0x7f7a1011c8a0 2024-09-15T04:05:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.610+0000 7f7a16044640 1 -- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a1011c8a0 msgr2=0x7f7a1011b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.610+0000 7f7a16044640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a1011c8a0 0x7f7a1011b910 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c010da0 tx=0x7f7a0c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.610+0000 7f7a067fc640 1 -- 172.21.15.110:0/3366138136 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c010ea0 con 0x7f7a1011c8a0 2024-09-15T04:05:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.610+0000 7f7a067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.610+0000 7f7a067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.610+0000 7f7a067fc640 10 monclient: dump: 2024-09-15T04:05:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.633 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.610+0000 7f7a15843640 1 --2- 172.21.15.110:0/3366138136 >> v2:172.21.15.177:6812/28201 conn(0x7f79f4038700 0x7f79f403abb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7a00034000 tx=0x7f7a00032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.610+0000 7f7a067fc640 1 -- 172.21.15.110:0/3366138136 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7a0c014640 con 0x7f7a1011c8a0 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.610+0000 7f7a067fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.610+0000 7f7a067fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.610+0000 7f7a067fc640 1 -- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a1011c8a0 msgr2=0x7f7a1011b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.610+0000 7f7a067fc640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a1011c8a0 0x7f7a1011b910 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c010da0 tx=0x7f7a0c008040 comp rx=0 tx=0).stop 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.610+0000 7f7a067fc640 1 -- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a1011c8a0 msgr2=0x7f7a1011b910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:05:03.634 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.610+0000 7f7a067fc640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a1011c8a0 0x7f7a1011b910 secure :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c010da0 tx=0x7f7a0c008040 comp rx=0 tx=0).stop 2024-09-15T04:05:03.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.615+0000 7f7a182a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:05:03.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.616+0000 7f7a182a8640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10130c20 0x7f7a1011be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.616+0000 7f7a182a8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.616+0000 7f7a182a8640 1 -- 172.21.15.110:0/3366138136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1011dc50 con 0x7f7a10130c20 2024-09-15T04:05:03.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.616+0000 7f7a16044640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10130c20 0x7f7a1011be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.616+0000 7f7a16044640 10 monclient: get_auth_request con 0x7f7a10130c20 auth_method 0 2024-09-15T04:05:03.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.616+0000 7f7a16044640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.638 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.616+0000 7f7a16044640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.616+0000 7f7a16044640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.616+0000 7f7a16044640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.616+0000 7f7a16044640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.616+0000 7f7a16044640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.616+0000 7f7a16044640 10 monclient(hunting): handle_auth_done global_id 55167 payload 824 2024-09-15T04:05:03.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.616+0000 7f7a16044640 1 -- 172.21.15.110:0/3366138136 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7a1009ba50 con 0x7f7a10130c20 2024-09-15T04:05:03.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.618+0000 7f7a16044640 1 -- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10130c20 msgr2=0x7f7a1011be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.618+0000 7f7a16044640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10130c20 0x7f7a1011be50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c010da0 tx=0x7f7a0c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.618+0000 7f7a067fc640 1 -- 172.21.15.110:0/3366138136 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c022070 con 0x7f7a10130c20 2024-09-15T04:05:03.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.618+0000 7f7a067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.618+0000 7f7a067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:03.639 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.618+0000 7f7a067fc640 10 monclient: dump: 2024-09-15T04:05:03.639 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.639 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.640 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.640 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.640 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.640 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.640 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.640 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.640 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.640 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.618+0000 7f7a067fc640 1 -- 172.21.15.110:0/3366138136 <== mon.0 v2:172.21.15.110:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4749 (secure 0 0 0) 0x7f7a0c0135e0 con 0x7f7a10130c20 2024-09-15T04:05:03.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.618+0000 7f7a067fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:05:03.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.618+0000 7f7a067fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:05:03.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.618+0000 7f7a067fc640 1 -- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10130c20 msgr2=0x7f7a1011be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.618+0000 7f7a067fc640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10130c20 0x7f7a1011be50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c010da0 tx=0x7f7a0c01b000 comp rx=0 tx=0).stop 2024-09-15T04:05:03.640 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.618+0000 7f7a067fc640 1 -- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10130c20 msgr2=0x7f7a1011be50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:05:03.641 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.618+0000 7f7a067fc640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10130c20 0x7f7a1011be50 secure :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c010da0 tx=0x7f7a0c01b000 comp rx=0 tx=0).stop 2024-09-15T04:05:03.641 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:05:03.641 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:05:03.641 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:05:03.641 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "leader", 2024-09-15T04:05:03.641 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 34, 2024-09-15T04:05:03.641 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:05:03.641 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:05:03.641 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:05:03.641 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:05:03.642 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.642 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 0, 2024-09-15T04:05:03.642 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:05:03.642 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:05:03.642 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:05:03.642 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:05:03.642 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:05:03.642 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:05:03.642 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:05:03.642 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:05:03.642 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:05:03.642 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:05:03.642 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:05:03.642 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:05:03.643 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.643 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:05:03.643 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:05:03.643 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:05:03.643 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:05:03.643 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:05:03.643 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:05:03.643 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:05:03.643 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:05:03.643 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:05:03.643 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:05:03.643 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:05:03.643 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.643 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.644 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:05:03.644 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:05:03.644 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:05:03.644 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:05:03.644 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:05:03.644 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:05:03.644 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:05:03.644 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:05:03.644 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:05:03.644 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:05:03.644 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:05:03.644 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:05:03.644 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:05:03.644 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:05:03.645 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:05:03.645 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:05:03.645 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:05:03.645 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:05:03.645 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:05:03.645 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:05:03.645 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:05:03.645 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:05:03.645 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:05:03.645 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:05:03.645 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:05:03.645 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:05:03.645 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.645 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:05:03.646 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.646 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:05:03.646 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.646 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:05:03.646 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:05:03.646 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:05:03.646 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:05:03.646 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.646 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:05:03.646 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:05:03.646 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.646 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.646 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.646 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:05:03.647 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:05:03.648 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:05:03.648 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.648 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:05:03.648 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:05:03.648 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.648 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.649 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.649 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:05:03.649 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:05:03.649 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.649 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.649 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.649 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.649 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:05:03.649 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:05:03.649 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:05:03.649 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:05:03.649 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:05:03.649 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.650 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.650 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:05:03.650 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:05:03.650 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:05:03.650 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:05:03.650 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.650 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:05:03.650 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:05:03.650 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.650 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.650 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.650 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:05:03.651 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:05:03.651 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.651 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.651 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.651 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.651 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:05:03.651 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:05:03.651 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:05:03.651 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:05:03.651 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:05:03.651 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.651 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.652 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.652 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:05:03.652 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:05:03.652 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.652 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:05:03.652 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:05:03.652 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:05:03.652 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.652 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.652 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T04:05:03.652 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.652 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:05:03.653 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:05:03.653 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 2 2024-09-15T04:05:03.653 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.653 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.653 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:05:03.653 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.653 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:05:03.653 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:05:03.653 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 2 2024-09-15T04:05:03.653 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.653 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.653 INFO:teuthology.orchestra.run.smithi110.stdout: "mgr": [ 2024-09-15T04:05:03.653 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.653 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:05:03.654 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:05:03.654 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:05:03.654 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.654 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.654 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.654 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:05:03.654 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:05:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.619+0000 7f7a182a8640 1 -- 172.21.15.110:0/3366138136 >> v2:172.21.15.177:6812/28201 conn(0x7f79f4038700 msgr2=0x7f79f403abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.619+0000 7f7a182a8640 1 --2- 172.21.15.110:0/3366138136 >> v2:172.21.15.177:6812/28201 conn(0x7f79f4038700 0x7f79f403abb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7a00034000 tx=0x7f7a00032000 comp rx=0 tx=0).stop 2024-09-15T04:05:03.655 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.619+0000 7f7a182a8640 10 monclient: shutdown 2024-09-15T04:05:03.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.619+0000 7f7a182a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:03.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.619+0000 7f7a182a8640 1 -- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10131a80 msgr2=0x7f7a1011b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.619+0000 7f7a182a8640 1 --2- 172.21.15.110:0/3366138136 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7a10131a80 0x7f7a1011b3d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7a0800b810 tx=0x7f7a080092b0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.619+0000 7f7a16845640 1 -- 172.21.15.110:0/3366138136 reap_dead start 2024-09-15T04:05:03.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.619+0000 7f7a16845640 1 -- 172.21.15.110:0/3366138136 reap_dead start 2024-09-15T04:05:03.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.619+0000 7f7a182a8640 1 -- 172.21.15.110:0/3366138136 shutdown_connections 2024-09-15T04:05:03.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.619+0000 7f7a182a8640 1 -- 172.21.15.110:0/3366138136 >> 172.21.15.110:0/3366138136 conn(0x7f7a10062f70 msgr2=0x7f7a100a2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:03.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.619+0000 7f7a182a8640 1 -- 172.21.15.110:0/3366138136 shutdown_connections 2024-09-15T04:05:03.656 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.619+0000 7f7a182a8640 1 -- 172.21.15.110:0/3366138136 wait complete. 2024-09-15T04:05:03.656 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:05:03.657 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T04:05:03.657 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T04:05:03.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.750+0000 7f8d1e300640 10 monclient: get_monmap_and_config 2024-09-15T04:05:03.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.750+0000 7f8d1e300640 10 monclient: build_initial_monmap 2024-09-15T04:05:03.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.750+0000 7f8d1e300640 10 monclient: monmap: 2024-09-15T04:05:03.751 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:03.751 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.751 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:03.751028+0000 2024-09-15T04:05:03.751 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:03.751028+0000 2024-09-15T04:05:03.751 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:03.751 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.751 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:03.751 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:03.752 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:03.752 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:03.752 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:03.752 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.750+0000 7f8d1e300640 1 Processor -- start 2024-09-15T04:05:03.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.750+0000 7f8d1e300640 1 -- start start 2024-09-15T04:05:03.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.750+0000 7f8d1e300640 10 monclient: init 2024-09-15T04:05:03.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d1e300640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:03.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d1e300640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-15T04:05:03.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d1e300640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d180a1a10 0x7f8d18131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d1e300640 10 monclient(hunting): picked mon.noname-b con 0x7f8d180a1a10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:03.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d1e300640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d1809e770 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:03.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d1e300640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d181369e0 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:03.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d1e300640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1809f290 con 0x7f8d181369e0 2024-09-15T04:05:03.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d1e300640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d180a0850 con 0x7f8d1809e770 2024-09-15T04:05:03.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d1e300640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d1e300640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1809c9a0 con 0x7f8d180a1a10 2024-09-15T04:05:03.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d1e300640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:03.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d1e300640 10 monclient(hunting): authenticate will time out at 1127.395413s 2024-09-15T04:05:03.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d180a1a10 0x7f8d18131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d1c89d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8d181369e0 0x7f8d18138dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58902/0 (socket says 172.21.15.110:58902) 2024-09-15T04:05:03.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d180a1a10 0x7f8d18131b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35042/0 (socket says 172.21.15.110:35042) 2024-09-15T04:05:03.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d1c89d640 1 -- 172.21.15.110:0/957503065 learned_addr learned my addr 172.21.15.110:0/957503065 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:03.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 10 monclient(hunting): get_auth_request con 0x7f8d180a1a10 auth_method 0 2024-09-15T04:05:03.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 10 monclient(hunting): handle_auth_done global_id 74105 payload 274 2024-09-15T04:05:03.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 1 -- 172.21.15.110:0/957503065 >> v1:172.21.15.177:6789/0 conn(0x7f8d1809e770 legacy=0x7f8d1809eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:03.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 1 -- 172.21.15.110:0/957503065 >> v1:172.21.15.110:6789/0 conn(0x7f8d181369e0 legacy=0x7f8d18138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:03.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 10 monclient: _finish_hunting 0 2024-09-15T04:05:03.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 1 monclient: found mon.noname-b 2024-09-15T04:05:03.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:03.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:05:03.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.751+0000 7f8d17fff640 1 -- 172.21.15.110:0/957503065 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1809cc80 con 0x7f8d180a1a10 2024-09-15T04:05:03.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d17fff640 1 -- 172.21.15.110:0/957503065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d180a1a10 msgr2=0x7f8d18131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d17fff640 1 --2- 172.21.15.110:0/957503065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d180a1a10 0x7f8d18131b90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009880 tx=0x7f8d080090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9f3e1416cc65cf4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d16ffd640 1 -- 172.21.15.110:0/957503065 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08032440 con 0x7f8d180a1a10 2024-09-15T04:05:03.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d16ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:05:03.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d16ffd640 10 monclient: dump: 2024-09-15T04:05:03.768 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.768 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.768 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.768 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.768 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.768 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.769 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.769 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.769 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.769 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d16ffd640 10 monclient: _finish_auth 0 2024-09-15T04:05:03.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:03.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d16ffd640 1 -- 172.21.15.110:0/957503065 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d080325a0 con 0x7f8d180a1a10 2024-09-15T04:05:03.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d1e300640 5 monclient: authenticate success, global_id 74105 2024-09-15T04:05:03.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d1e300640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:03.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d16ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:03.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d1e300640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:03.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d16ffd640 1 -- 172.21.15.110:0/957503065 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08032940 con 0x7f8d180a1a10 2024-09-15T04:05:03.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d1e300640 4 set_mon_vals no callback set 2024-09-15T04:05:03.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:03.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.754+0000 7f8d16ffd640 10 monclient: dump: 2024-09-15T04:05:03.770 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.770 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.770 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.770 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.770 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.770 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.770 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.771 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.771 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.771 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.755+0000 7f8d1e300640 10 monclient: shutdown 2024-09-15T04:05:03.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.755+0000 7f8d1e300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:03.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.755+0000 7f8d1e300640 1 -- 172.21.15.110:0/957503065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d180a1a10 msgr2=0x7f8d18131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.755+0000 7f8d1e300640 1 --2- 172.21.15.110:0/957503065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d180a1a10 0x7f8d18131b90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009880 tx=0x7f8d080090f0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.755+0000 7f8d1e300640 1 -- 172.21.15.110:0/957503065 shutdown_connections 2024-09-15T04:05:03.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.755+0000 7f8d1e300640 1 --2- 172.21.15.110:0/957503065 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d180a1a10 0x7f8d18131b90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.755+0000 7f8d1e300640 1 -- 172.21.15.110:0/957503065 >> 172.21.15.110:0/957503065 conn(0x7f8d1806e860 msgr2=0x7f8d1806ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:03.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.755+0000 7f8d1e300640 1 -- 172.21.15.110:0/957503065 shutdown_connections 2024-09-15T04:05:03.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.755+0000 7f8d1e300640 1 -- 172.21.15.110:0/957503065 wait complete. 2024-09-15T04:05:03.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.755+0000 7f8d1e300640 10 monclient: build_initial_monmap 2024-09-15T04:05:03.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.755+0000 7f8d1e300640 10 monclient: monmap: 2024-09-15T04:05:03.772 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:03.772 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:03.772 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:03.772 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:03.772 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:03.772 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.772 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:03.772 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:03.772 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:03.772 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 1 Processor -- start 2024-09-15T04:05:03.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 1 -- start start 2024-09-15T04:05:03.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 10 monclient: init 2024-09-15T04:05:03.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:03.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T04:05:03.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d1809e770 0x7f8d1809a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 10 monclient(hunting): picked mon.noname-b con 0x7f8d1809e770 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:03.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d180a1a10 0x7f8d180960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 10 monclient(hunting): picked mon.noname-a con 0x7f8d180a1a10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:03.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d181369e0 0x7f8d18096600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 10 monclient(hunting): picked mon.noname-c con 0x7f8d181369e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:03.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180a1230 con 0x7f8d180a1a10 2024-09-15T04:05:03.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180a2eb0 con 0x7f8d1809e770 2024-09-15T04:05:03.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d18099a40 con 0x7f8d181369e0 2024-09-15T04:05:03.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:03.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1e300640 10 monclient(hunting): authenticate will time out at 1127.400786s 2024-09-15T04:05:03.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1c89d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d181369e0 0x7f8d18096600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d177fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d180a1a10 0x7f8d180960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1c89d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d181369e0 0x7f8d18096600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35050/0 (socket says 172.21.15.110:35050) 2024-09-15T04:05:03.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d177fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d180a1a10 0x7f8d180960c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42728/0 (socket says 172.21.15.110:42728) 2024-09-15T04:05:03.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1c89d640 1 -- 172.21.15.110:0/807461628 learned_addr learned my addr 172.21.15.110:0/807461628 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:03.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1c89d640 10 monclient(hunting): get_auth_request con 0x7f8d181369e0 auth_method 0 2024-09-15T04:05:03.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1c89d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1c89d640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1c89d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d177fe640 10 monclient(hunting): get_auth_request con 0x7f8d180a1a10 auth_method 0 2024-09-15T04:05:03.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1c89d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1c89d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.756+0000 7f8d1c89d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d17fff640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d1809e770 0x7f8d1809a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d17fff640 10 monclient(hunting): get_auth_request con 0x7f8d1809e770 auth_method 0 2024-09-15T04:05:03.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d17fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d1c89d640 10 monclient(hunting): handle_auth_done global_id 74108 payload 824 2024-09-15T04:05:03.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d1c89d640 1 -- 172.21.15.110:0/807461628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d1809e770 msgr2=0x7f8d1809a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d1c89d640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d1809e770 0x7f8d1809a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d1c89d640 1 -- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d180a1a10 msgr2=0x7f8d180960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d1c89d640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d180a1a10 0x7f8d180960c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d1c89d640 10 monclient: _finish_hunting 0 2024-09-15T04:05:03.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d1c89d640 1 monclient: found mon.noname-c 2024-09-15T04:05:03.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d1c89d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:03.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d1c89d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:03.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d1c89d640 1 -- 172.21.15.110:0/807461628 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d08009390 con 0x7f8d181369e0 2024-09-15T04:05:03.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d17fff640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8d1809e770 0x7f8d1809a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T04:05:03.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.757+0000 7f8d177fe640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8d180a1a10 0x7f8d180960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:03.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8d1c89d640 1 -- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d181369e0 msgr2=0x7f8d18096600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8d1c89d640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d181369e0 0x7f8d18096600 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000ccc0 tx=0x7f8d10005370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8cfffff640 1 -- 172.21.15.110:0/807461628 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1000ce80 con 0x7f8d181369e0 2024-09-15T04:05:03.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8cfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8cfffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:03.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8cfffff640 10 monclient: dump: 2024-09-15T04:05:03.777 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.777 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.777 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.777 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.777 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8cfffff640 10 monclient: _finish_auth 0 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8cfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8d1e300640 5 monclient: authenticate success, global_id 74108 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8d1e300640 10 monclient: _renew_subs 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8cfffff640 1 -- 172.21.15.110:0/807461628 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1000aa60 con 0x7f8d181369e0 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8d1e300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8d1e300640 1 -- 172.21.15.110:0/807461628 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d18099760 con 0x7f8d181369e0 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8cfffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8cfffff640 1 -- 172.21.15.110:0/807461628 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1000ae20 con 0x7f8d181369e0 2024-09-15T04:05:03.778 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8cfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8cfffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:03.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8cfffff640 10 monclient: dump: 2024-09-15T04:05:03.779 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.779 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.779 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.779 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.779 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.779 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.779 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.779 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.779 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.779 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8d1e300640 10 monclient: _renew_subs 2024-09-15T04:05:03.779 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8d167fc640 4 set_mon_vals no callback set 2024-09-15T04:05:03.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8d1e300640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:03.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.759+0000 7f8d1e300640 1 -- 172.21.15.110:0/807461628 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d1809a960 con 0x7f8d181369e0 2024-09-15T04:05:03.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d1e300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:03.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d1e300640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18096b40 0x7f8d181326c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d1e300640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d1e300640 1 -- 172.21.15.110:0/807461628 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d18063e80 con 0x7f8d18096b40 2024-09-15T04:05:03.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8cfffff640 1 -- 172.21.15.110:0/807461628 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f8d10007300 con 0x7f8d181369e0 2024-09-15T04:05:03.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d17fff640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18096b40 0x7f8d181326c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d17fff640 10 monclient: get_auth_request con 0x7f8d18096b40 auth_method 0 2024-09-15T04:05:03.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d17fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8cfffff640 1 --2- 172.21.15.110:0/807461628 >> v2:172.21.15.177:6812/28201 conn(0x7f8d04038730 0x7f8d0403abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d17fff640 10 monclient(hunting): handle_auth_done global_id 74111 payload 824 2024-09-15T04:05:03.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d17fff640 1 -- 172.21.15.110:0/807461628 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d1809a1b0 con 0x7f8d18096b40 2024-09-15T04:05:03.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8cfffff640 1 -- 172.21.15.110:0/807461628 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8d1004b700 con 0x7f8d181369e0 2024-09-15T04:05:03.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d177fe640 1 --2- 172.21.15.110:0/807461628 >> v2:172.21.15.177:6812/28201 conn(0x7f8d04038730 0x7f8d0403abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.760+0000 7f8d177fe640 10 monclient: get_auth_request con 0x7f8d04038730 auth_method 0 2024-09-15T04:05:03.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.762+0000 7f8d17fff640 1 -- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18096b40 msgr2=0x7f8d181326c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.762+0000 7f8d17fff640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18096b40 0x7f8d181326c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8d08034000 tx=0x7f8d08032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.762+0000 7f8cfffff640 1 -- 172.21.15.110:0/807461628 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08031a00 con 0x7f8d18096b40 2024-09-15T04:05:03.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.762+0000 7f8cfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.763+0000 7f8cfffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:03.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.763+0000 7f8cfffff640 10 monclient: dump: 2024-09-15T04:05:03.782 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.782 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.782 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.782 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.782 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.782 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.782 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.783 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.783 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.783 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.763+0000 7f8cfffff640 1 -- 172.21.15.110:0/807461628 <== mon.2 v2:172.21.15.110:3301/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f8d08031b60 con 0x7f8d18096b40 2024-09-15T04:05:03.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.763+0000 7f8cfffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:03.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.763+0000 7f8cfffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:03.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.763+0000 7f8cfffff640 1 -- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18096b40 msgr2=0x7f8d181326c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.763+0000 7f8cfffff640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18096b40 0x7f8d181326c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8d08034000 tx=0x7f8d08032000 comp rx=0 tx=0).stop 2024-09-15T04:05:03.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.763+0000 7f8cfffff640 1 -- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18096b40 msgr2=0x7f8d181326c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:05:03.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.763+0000 7f8cfffff640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18096b40 0x7f8d181326c0 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8d08034000 tx=0x7f8d08032000 comp rx=0 tx=0).stop 2024-09-15T04:05:03.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.763+0000 7f8d177fe640 1 --2- 172.21.15.110:0/807461628 >> v2:172.21.15.177:6812/28201 conn(0x7f8d04038730 0x7f8d0403abe0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c009e00 tx=0x7f8d0c006f30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.768+0000 7f8d1e300640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:05:03.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.768+0000 7f8d1e300640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18132c00 0x7f8d18133000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.768+0000 7f8d1e300640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.768+0000 7f8d1e300640 1 -- 172.21.15.110:0/807461628 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d180998f0 con 0x7f8d18132c00 2024-09-15T04:05:03.785 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.769+0000 7f8d17fff640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18132c00 0x7f8d18133000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.769+0000 7f8d17fff640 10 monclient: get_auth_request con 0x7f8d18132c00 auth_method 0 2024-09-15T04:05:03.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.769+0000 7f8d17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.769+0000 7f8d17fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.769+0000 7f8d17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.769+0000 7f8d17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.769+0000 7f8d17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.769+0000 7f8d17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.769+0000 7f8d17fff640 10 monclient(hunting): handle_auth_done global_id 74114 payload 824 2024-09-15T04:05:03.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.769+0000 7f8d17fff640 1 -- 172.21.15.110:0/807461628 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8d180647e0 con 0x7f8d18132c00 2024-09-15T04:05:03.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.771+0000 7f8d17fff640 1 -- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18132c00 msgr2=0x7f8d18133000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.771+0000 7f8d17fff640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18132c00 0x7f8d18133000 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8d080097d0 tx=0x7f8d0803c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.771+0000 7f8cfffff640 1 -- 172.21.15.110:0/807461628 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08048a50 con 0x7f8d18132c00 2024-09-15T04:05:03.786 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.771+0000 7f8cfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.771+0000 7f8cfffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.771+0000 7f8cfffff640 10 monclient: dump: 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.771+0000 7f8cfffff640 1 -- 172.21.15.110:0/807461628 <== mon.2 v2:172.21.15.110:3301/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4407 (secure 0 0 0) 0x7f8d08048bb0 con 0x7f8d18132c00 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.771+0000 7f8cfffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:05:03.787 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.771+0000 7f8cfffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:05:03.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.771+0000 7f8cfffff640 1 -- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18132c00 msgr2=0x7f8d18133000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.771+0000 7f8cfffff640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18132c00 0x7f8d18133000 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8d080097d0 tx=0x7f8d0803c000 comp rx=0 tx=0).stop 2024-09-15T04:05:03.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.771+0000 7f8cfffff640 1 -- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18132c00 msgr2=0x7f8d18133000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:05:03.788 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.771+0000 7f8cfffff640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d18132c00 0x7f8d18133000 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8d080097d0 tx=0x7f8d0803c000 comp rx=0 tx=0).stop 2024-09-15T04:05:03.788 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:05:03.788 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:05:03.788 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:05:03.788 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T04:05:03.788 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 34, 2024-09-15T04:05:03.788 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:05:03.789 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:05:03.789 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:05:03.789 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:05:03.789 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.789 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 0, 2024-09-15T04:05:03.789 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:05:03.789 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:05:03.789 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:05:03.789 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:05:03.789 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:05:03.789 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:05:03.789 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:05:03.789 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:05:03.790 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:05:03.791 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:05:03.792 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:05:03.792 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:05:03.792 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:05:03.792 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:05:03.792 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:05:03.792 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:05:03.792 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:05:03.792 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:05:03.792 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:05:03.792 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:05:03.792 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:05:03.792 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:05:03.793 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:05:03.793 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:05:03.793 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.793 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:05:03.793 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.793 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:05:03.793 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.793 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:05:03.793 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:05:03.793 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:05:03.793 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:05:03.793 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.793 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:05:03.794 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:05:03.794 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.794 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.794 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.794 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:05:03.794 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:05:03.794 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.794 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.794 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.794 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.794 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:05:03.794 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:05:03.794 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:05:03.795 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:05:03.795 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:05:03.795 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.795 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.795 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:05:03.795 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:05:03.795 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:05:03.795 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:05:03.795 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.795 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:05:03.795 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:05:03.795 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.795 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.795 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.796 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:05:03.796 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:05:03.796 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.796 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.796 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.796 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.796 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:05:03.796 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:05:03.796 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:05:03.796 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:05:03.796 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:05:03.796 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.797 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.797 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:05:03.797 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:05:03.797 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:05:03.797 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:05:03.797 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.797 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:05:03.797 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:05:03.797 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.797 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.797 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.797 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:05:03.797 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:05:03.797 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.798 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.798 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.798 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.798 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:05:03.798 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:05:03.798 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:05:03.798 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:05:03.798 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:05:03.798 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.798 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.798 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.798 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:05:03.798 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:05:03.799 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.799 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:05:03.799 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:05:03.799 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:05:03.799 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.799 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.799 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:05:03.799 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.799 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:05:03.799 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:05:03.799 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 2 2024-09-15T04:05:03.799 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.799 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.800 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.800 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:05:03.800 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:05:03.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.772+0000 7f8d1e300640 1 -- 172.21.15.110:0/807461628 >> v2:172.21.15.177:6812/28201 conn(0x7f8d04038730 msgr2=0x7f8d0403abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.772+0000 7f8d1e300640 1 --2- 172.21.15.110:0/807461628 >> v2:172.21.15.177:6812/28201 conn(0x7f8d04038730 0x7f8d0403abe0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c009e00 tx=0x7f8d0c006f30 comp rx=0 tx=0).stop 2024-09-15T04:05:03.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.772+0000 7f8d1e300640 10 monclient: shutdown 2024-09-15T04:05:03.801 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.772+0000 7f8d1e300640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:03.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.772+0000 7f8d1e300640 1 -- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d181369e0 msgr2=0x7f8d18096600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.772+0000 7f8d1e300640 1 --2- 172.21.15.110:0/807461628 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f8d181369e0 0x7f8d18096600 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000ccc0 tx=0x7f8d10005370 comp rx=0 tx=0).stop 2024-09-15T04:05:03.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.772+0000 7f8d1c89d640 1 -- 172.21.15.110:0/807461628 reap_dead start 2024-09-15T04:05:03.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.772+0000 7f8d1c89d640 1 -- 172.21.15.110:0/807461628 reap_dead start 2024-09-15T04:05:03.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.772+0000 7f8d1e300640 1 -- 172.21.15.110:0/807461628 shutdown_connections 2024-09-15T04:05:03.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.772+0000 7f8d1e300640 1 -- 172.21.15.110:0/807461628 >> 172.21.15.110:0/807461628 conn(0x7f8d1806e860 msgr2=0x7f8d18139c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:03.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.772+0000 7f8d1e300640 1 -- 172.21.15.110:0/807461628 shutdown_connections 2024-09-15T04:05:03.802 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.772+0000 7f8d1e300640 1 -- 172.21.15.110:0/807461628 wait complete. 2024-09-15T04:05:03.802 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:05:03.802 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T04:05:03.803 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T04:05:03.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.895+0000 7f7b9219d640 10 monclient: get_monmap_and_config 2024-09-15T04:05:03.895 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.895+0000 7f7b9219d640 10 monclient: build_initial_monmap 2024-09-15T04:05:03.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.895+0000 7f7b9219d640 10 monclient: monmap: 2024-09-15T04:05:03.896 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:03.896 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.896 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:03.896064+0000 2024-09-15T04:05:03.896 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:03.896064+0000 2024-09-15T04:05:03.896 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:03.896 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.896 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:03.896 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:03.896 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:03.896 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:03.896 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:03.897 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.895+0000 7f7b9219d640 1 Processor -- start 2024-09-15T04:05:03.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.895+0000 7f7b9219d640 1 -- start start 2024-09-15T04:05:03.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.895+0000 7f7b9219d640 10 monclient: init 2024-09-15T04:05:03.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b9219d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:03.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b9219d640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-15T04:05:03.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b9219d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b8c0a1cf0 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:03.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b9219d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7b8c131a80 0x7f7b8c09ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b9219d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b8c131a80 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:03.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b9219d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c136940 0x7f7b8c138d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b9219d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b8c136940 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:03.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b9219d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b8c09cb70 con 0x7f7b8c0a1cf0 2024-09-15T04:05:03.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b9219d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b9219d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8c0a20c0 con 0x7f7b8c131a80 2024-09-15T04:05:03.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b9219d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b9219d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c09c330 con 0x7f7b8c136940 2024-09-15T04:05:03.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b9219d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:03.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b9219d640 10 monclient(hunting): authenticate will time out at 1127.540444s 2024-09-15T04:05:03.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8affd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7b8c131a80 0x7f7b8c09ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8b7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f7b8c0a1cf0 0x7f7b8c09e660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58908/0 (socket says 172.21.15.110:58908) 2024-09-15T04:05:03.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8b7fe640 1 -- 172.21.15.110:0/237209482 learned_addr learned my addr 172.21.15.110:0/237209482 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:03.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8affd640 10 monclient(hunting): get_auth_request con 0x7f7b8c131a80 auth_method 0 2024-09-15T04:05:03.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 1 --2- 172.21.15.110:0/237209482 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c136940 0x7f7b8c138d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 10 monclient(hunting): get_auth_request con 0x7f7b8c136940 auth_method 0 2024-09-15T04:05:03.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_done global_id 74117 payload 274 2024-09-15T04:05:03.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 1 -- 172.21.15.110:0/237209482 >> v2:172.21.15.110:3300/0 conn(0x7f7b8c131a80 msgr2=0x7f7b8c09ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:03.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 1 --2- 172.21.15.110:0/237209482 >> v2:172.21.15.110:3300/0 conn(0x7f7b8c131a80 0x7f7b8c09ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 1 -- 172.21.15.110:0/237209482 >> v1:172.21.15.110:6789/0 conn(0x7f7b8c0a1cf0 legacy=0x7f7b8c09e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:03.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 10 monclient: _finish_hunting 0 2024-09-15T04:05:03.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 1 monclient: found mon.noname-b 2024-09-15T04:05:03.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:03.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:05:03.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8bfff640 1 -- 172.21.15.110:0/237209482 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8c131fb0 con 0x7f7b8c136940 2024-09-15T04:05:03.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.896+0000 7f7b8affd640 1 --2- 172.21.15.110:0/237209482 >> v2:172.21.15.110:3300/0 conn(0x7f7b8c131a80 0x7f7b8c09ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:03.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8bfff640 1 -- 172.21.15.110:0/237209482 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c136940 msgr2=0x7f7b8c138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8bfff640 1 --2- 172.21.15.110:0/237209482 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c136940 0x7f7b8c138d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009e60 tx=0x7f7b84009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b549cfb82ca43bfb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8bfff640 1 -- 172.21.15.110:0/237209482 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c136940 msgr2=0x7f7b8c138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-15T04:05:03.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8bfff640 1 -- 172.21.15.110:0/237209482 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c136940 msgr2=0x7f7b8c138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T04:05:03.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8bfff640 1 --2- 172.21.15.110:0/237209482 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c136940 0x7f7b8c138d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009e60 tx=0x7f7b84009780 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-15T04:05:03.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8bfff640 1 --2- 172.21.15.110:0/237209482 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c136940 0x7f7b8c138d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009e60 tx=0x7f7b84009780 comp rx=0 tx=0).stop 2024-09-15T04:05:03.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8a7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:03.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8a7fc640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:03.911 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8a7fc640 1 -- 172.21.15.110:0/237209482 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c136940 msgr2=0x7f7b8c138d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:05:03.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8a7fc640 1 --2- 172.21.15.110:0/237209482 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c136940 0x7f7b8c138d20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8a7fc640 10 monclient: _add_conns ranks=[1,4] 2024-09-15T04:05:03.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8a7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b740034c0 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:03.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8a7fc640 1 --2- 172.21.15.110:0/237209482 >> v2:172.21.15.177:3300/0 conn(0x7f7b74007440 0x7f7b740098f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f7b74007440 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:03.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8a7fc640 1 -- 172.21.15.110:0/237209482 --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7400a1c0 con 0x7f7b740034c0 2024-09-15T04:05:03.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8a7fc640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8a7fc640 1 -- 172.21.15.110:0/237209482 --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b7400b3c0 con 0x7f7b74007440 2024-09-15T04:05:03.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8a7fc640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:03.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.899+0000 7f7b8a7fc640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:03.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 1 --2- 172.21.15.110:0/237209482 >> v2:172.21.15.177:3300/0 conn(0x7f7b74007440 0x7f7b740098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 10 monclient(hunting): get_auth_request con 0x7f7b74007440 auth_method 0 2024-09-15T04:05:03.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.912 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T04:05:03.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-15T04:05:03.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 10 monclient(hunting): handle_auth_done global_id 74117 payload 293 2024-09-15T04:05:03.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 1 -- 172.21.15.110:0/237209482 >> v1:172.21.15.177:6789/0 conn(0x7f7b740034c0 legacy=0x7f7b74005930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:03.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 10 monclient: _finish_hunting 0 2024-09-15T04:05:03.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 1 monclient: found mon.noname-c 2024-09-15T04:05:03.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:03.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T04:05:03.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.900+0000 7f7b8affd640 1 -- 172.21.15.110:0/237209482 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7400b5f0 con 0x7f7b74007440 2024-09-15T04:05:03.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.902+0000 7f7b8affd640 1 -- 172.21.15.110:0/237209482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b74007440 msgr2=0x7f7b740098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.913 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.902+0000 7f7b8affd640 1 --2- 172.21.15.110:0/237209482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b74007440 0x7f7b740098f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7b80006e70 tx=0x7f7b800090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=839561fa7a53d46d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b8affd640 1 --2- 172.21.15.110:0/237209482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b74007440 0x7f7b740098f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7b80006e70 tx=0x7f7b800090f0 comp rx=0 tx=0).queue_received will delay after 1.72637e+09 on 0x7f7b800096b0 config(0 keys) v1 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b8a7fc640 1 -- 172.21.15.110:0/237209482 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8000ee70 con 0x7f7b74007440 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b8a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b8a7fc640 10 monclient: dump: 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.914 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b8a7fc640 10 monclient: _finish_auth 0 2024-09-15T04:05:03.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:03.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b8a7fc640 1 -- 172.21.15.110:0/237209482 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b800096b0 con 0x7f7b74007440 2024-09-15T04:05:03.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:03.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b9219d640 5 monclient: authenticate success, global_id 74117 2024-09-15T04:05:03.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b9219d640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:03.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b9219d640 4 set_mon_vals no callback set 2024-09-15T04:05:03.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b9219d640 10 monclient: shutdown 2024-09-15T04:05:03.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b9219d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:03.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b9219d640 1 -- 172.21.15.110:0/237209482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b74007440 msgr2=0x7f7b740098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b9219d640 1 --2- 172.21.15.110:0/237209482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b74007440 0x7f7b740098f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7b80006e70 tx=0x7f7b800090f0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b8bfff640 1 -- 172.21.15.110:0/237209482 reap_dead start 2024-09-15T04:05:03.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b8a7fc640 1 -- 172.21.15.110:0/237209482 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 0 0 0) 0x7f7b800136c0 con 0x7f7b74007440 2024-09-15T04:05:03.915 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b8a7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T04:05:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b9219d640 1 -- 172.21.15.110:0/237209482 shutdown_connections 2024-09-15T04:05:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b9219d640 1 --2- 172.21.15.110:0/237209482 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b74007440 0x7f7b740098f0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b9219d640 1 -- 172.21.15.110:0/237209482 >> 172.21.15.110:0/237209482 conn(0x7f7b8c062f70 msgr2=0x7f7b8c063340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b9219d640 1 -- 172.21.15.110:0/237209482 shutdown_connections 2024-09-15T04:05:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.903+0000 7f7b9219d640 1 -- 172.21.15.110:0/237209482 wait complete. 2024-09-15T04:05:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.904+0000 7f7b9219d640 10 monclient: build_initial_monmap 2024-09-15T04:05:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.904+0000 7f7b9219d640 10 monclient: monmap: 2024-09-15T04:05:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:03.916 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:03.917 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.904+0000 7f7b9219d640 1 Processor -- start 2024-09-15T04:05:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.904+0000 7f7b9219d640 1 -- start start 2024-09-15T04:05:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.904+0000 7f7b9219d640 10 monclient: init 2024-09-15T04:05:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.904+0000 7f7b9219d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T04:05:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b74007440 0x7f7b8c140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b74007440 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c11b970 0x7f7b8c1307c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.917 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b8c11b970 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:03.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b8c130d00 0x7f7b8c099a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b8c130d00 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:03.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c09b9b0 con 0x7f7b8c130d00 2024-09-15T04:05:03.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c11cbb0 con 0x7f7b74007440 2024-09-15T04:05:03.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c141d20 con 0x7f7b8c11b970 2024-09-15T04:05:03.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:03.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b9219d640 10 monclient(hunting): authenticate will time out at 1127.549068s 2024-09-15T04:05:03.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b8c130d00 0x7f7b8c099a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8affd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c11b970 0x7f7b8c1307c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.918 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b8c130d00 0x7f7b8c099a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42758/0 (socket says 172.21.15.110:42758) 2024-09-15T04:05:03.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 1 -- 172.21.15.110:0/3356766683 learned_addr learned my addr 172.21.15.110:0/3356766683 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:03.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8affd640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c11b970 0x7f7b8c1307c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35096/0 (socket says 172.21.15.110:35096) 2024-09-15T04:05:03.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 10 monclient(hunting): get_auth_request con 0x7f7b8c130d00 auth_method 0 2024-09-15T04:05:03.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8affd640 10 monclient(hunting): get_auth_request con 0x7f7b8c11b970 auth_method 0 2024-09-15T04:05:03.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8b7fe640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b74007440 0x7f7b8c140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.919 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8b7fe640 10 monclient(hunting): get_auth_request con 0x7f7b74007440 auth_method 0 2024-09-15T04:05:03.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 10 monclient(hunting): handle_auth_done global_id 55176 payload 824 2024-09-15T04:05:03.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 1 -- 172.21.15.110:0/3356766683 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c11b970 msgr2=0x7f7b8c1307c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7b8c11b970 0x7f7b8c1307c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.920 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 1 -- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b74007440 msgr2=0x7f7b8c140e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b74007440 0x7f7b8c140e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:03.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 10 monclient: _finish_hunting 0 2024-09-15T04:05:03.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 1 monclient: found mon.noname-a 2024-09-15T04:05:03.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:03.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:03.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.905+0000 7f7b8bfff640 1 -- 172.21.15.110:0/3356766683 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8c09b510 con 0x7f7b8c130d00 2024-09-15T04:05:03.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b8bfff640 1 -- 172.21.15.110:0/3356766683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b8c130d00 msgr2=0x7f7b8c099a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b8bfff640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b8c130d00 0x7f7b8c099a90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7b840099f0 tx=0x7f7b84037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b737fe640 1 -- 172.21.15.110:0/3356766683 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b84009460 con 0x7f7b8c130d00 2024-09-15T04:05:03.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:03.921 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b737fe640 10 monclient: dump: 2024-09-15T04:05:03.921 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b737fe640 10 monclient: _finish_auth 0 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b737fe640 1 -- 172.21.15.110:0/3356766683 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8402d050 con 0x7f7b8c130d00 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b9219d640 5 monclient: authenticate success, global_id 55176 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b737fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:03.922 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b9219d640 10 monclient: _renew_subs 2024-09-15T04:05:03.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b737fe640 1 -- 172.21.15.110:0/3356766683 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8403d430 con 0x7f7b8c130d00 2024-09-15T04:05:03.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b9219d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:03.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b9219d640 1 -- 172.21.15.110:0/3356766683 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8c11c8d0 con 0x7f7b8c130d00 2024-09-15T04:05:03.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:03.923 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b737fe640 10 monclient: dump: 2024-09-15T04:05:03.923 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.923 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.923 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.923 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.923 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.923 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.923 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.923 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.924 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.924 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.907+0000 7f7b89ffb640 4 set_mon_vals no callback set 2024-09-15T04:05:03.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.908+0000 7f7b9219d640 10 monclient: _renew_subs 2024-09-15T04:05:03.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.908+0000 7f7b9219d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:03.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.908+0000 7f7b9219d640 1 -- 172.21.15.110:0/3356766683 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8c099fd0 con 0x7f7b8c130d00 2024-09-15T04:05:03.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.908+0000 7f7b9219d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:03.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.908+0000 7f7b9219d640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a280 0x7f7b8c095a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.908+0000 7f7b737fe640 1 -- 172.21.15.110:0/3356766683 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f7b84003040 con 0x7f7b8c130d00 2024-09-15T04:05:03.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.908+0000 7f7b9219d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.908+0000 7f7b9219d640 1 -- 172.21.15.110:0/3356766683 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c06f1e0 con 0x7f7b8c09a280 2024-09-15T04:05:03.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.908+0000 7f7b737fe640 1 --2- 172.21.15.110:0/3356766683 >> v2:172.21.15.177:6812/28201 conn(0x7f7b78038730 0x7f7b7803abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.909+0000 7f7b737fe640 1 -- 172.21.15.110:0/3356766683 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7b8406cf90 con 0x7f7b8c130d00 2024-09-15T04:05:03.924 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.909+0000 7f7b8b7fe640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a280 0x7f7b8c095a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.909+0000 7f7b8b7fe640 10 monclient: get_auth_request con 0x7f7b8c09a280 auth_method 0 2024-09-15T04:05:03.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.909+0000 7f7b8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.909+0000 7f7b8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.909+0000 7f7b8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.909+0000 7f7b8affd640 1 --2- 172.21.15.110:0/3356766683 >> v2:172.21.15.177:6812/28201 conn(0x7f7b78038730 0x7f7b7803abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.909+0000 7f7b8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.909+0000 7f7b8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.909+0000 7f7b8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.909+0000 7f7b8affd640 10 monclient: get_auth_request con 0x7f7b78038730 auth_method 0 2024-09-15T04:05:03.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.909+0000 7f7b8b7fe640 10 monclient(hunting): handle_auth_done global_id 5704 payload 824 2024-09-15T04:05:03.925 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.909+0000 7f7b8b7fe640 1 -- 172.21.15.110:0/3356766683 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7b8c0647e0 con 0x7f7b8c09a280 2024-09-15T04:05:03.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.911+0000 7f7b8b7fe640 1 -- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a280 msgr2=0x7f7b8c095a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.911+0000 7f7b8b7fe640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a280 0x7f7b8c095a70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00e740 tx=0x7f7b7c007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.911+0000 7f7b8affd640 1 --2- 172.21.15.110:0/3356766683 >> v2:172.21.15.177:6812/28201 conn(0x7f7b78038730 0x7f7b7803abe0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7b80009130 tx=0x7f7b80019000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.911+0000 7f7b737fe640 1 -- 172.21.15.110:0/3356766683 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c005050 con 0x7f7b8c09a280 2024-09-15T04:05:03.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.911+0000 7f7b737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.911+0000 7f7b737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:05:03.928 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.911+0000 7f7b737fe640 10 monclient: dump: 2024-09-15T04:05:03.928 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.929 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.929 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.929 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.929 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.929 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.929 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.929 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.929 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.929 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.912+0000 7f7b737fe640 1 -- 172.21.15.110:0/3356766683 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (secure 0 0 0) 0x7f7b7c0051b0 con 0x7f7b8c09a280 2024-09-15T04:05:03.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.912+0000 7f7b737fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:03.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.912+0000 7f7b737fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:03.929 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.912+0000 7f7b737fe640 1 -- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a280 msgr2=0x7f7b8c095a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.912+0000 7f7b737fe640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a280 0x7f7b8c095a70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00e740 tx=0x7f7b7c007f90 comp rx=0 tx=0).stop 2024-09-15T04:05:03.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.912+0000 7f7b737fe640 1 -- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a280 msgr2=0x7f7b8c095a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:05:03.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.912+0000 7f7b737fe640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a280 0x7f7b8c095a70 secure :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00e740 tx=0x7f7b7c007f90 comp rx=0 tx=0).stop 2024-09-15T04:05:03.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.917+0000 7f7b9219d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T04:05:03.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.917+0000 7f7b9219d640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a650 0x7f7b8c095fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:03.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.917+0000 7f7b9219d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:03.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.917+0000 7f7b9219d640 1 -- 172.21.15.110:0/3356766683 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c11ca60 con 0x7f7b8c09a650 2024-09-15T04:05:03.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.918+0000 7f7b8b7fe640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a650 0x7f7b8c095fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:03.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.918+0000 7f7b8b7fe640 10 monclient: get_auth_request con 0x7f7b8c09a650 auth_method 0 2024-09-15T04:05:03.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.918+0000 7f7b8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:03.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.918+0000 7f7b8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:03.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.918+0000 7f7b8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:03.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.918+0000 7f7b8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:03.930 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.918+0000 7f7b8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:03.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.918+0000 7f7b8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:03.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.918+0000 7f7b8b7fe640 10 monclient(hunting): handle_auth_done global_id 5707 payload 824 2024-09-15T04:05:03.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.918+0000 7f7b8b7fe640 1 -- 172.21.15.110:0/3356766683 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7b8c067130 con 0x7f7b8c09a650 2024-09-15T04:05:03.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.920+0000 7f7b8b7fe640 1 -- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a650 msgr2=0x7f7b8c095fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:03.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.920+0000 7f7b8b7fe640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a650 0x7f7b8c095fb0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00e740 tx=0x7f7b7c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:03.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b737fe640 1 -- 172.21.15.110:0/3356766683 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c021600 con 0x7f7b8c09a650 2024-09-15T04:05:03.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:03.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:05:03.931 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b737fe640 10 monclient: dump: 2024-09-15T04:05:03.931 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:03.931 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:03.931 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.932 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:03.932 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:03.932 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:03.932 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:03.932 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:03.932 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:03.932 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:03.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b737fe640 1 -- 172.21.15.110:0/3356766683 <== mon.1 v2:172.21.15.177:3300/0 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4577 (secure 0 0 0) 0x7f7b7c021760 con 0x7f7b8c09a650 2024-09-15T04:05:03.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b737fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T04:05:03.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b737fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:05:03.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b737fe640 1 -- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a650 msgr2=0x7f7b8c095fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b737fe640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a650 0x7f7b8c095fb0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00e740 tx=0x7f7b7c016000 comp rx=0 tx=0).stop 2024-09-15T04:05:03.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b737fe640 1 -- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a650 msgr2=0x7f7b8c095fb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T04:05:03.932 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b737fe640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b8c09a650 0x7f7b8c095fb0 secure :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00e740 tx=0x7f7b7c016000 comp rx=0 tx=0).stop 2024-09-15T04:05:03.933 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:05:03.933 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:05:03.933 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:05:03.933 INFO:teuthology.orchestra.run.smithi110.stdout: "state": "peon", 2024-09-15T04:05:03.933 INFO:teuthology.orchestra.run.smithi110.stdout: "election_epoch": 34, 2024-09-15T04:05:03.933 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum": [ 2024-09-15T04:05:03.933 INFO:teuthology.orchestra.run.smithi110.stdout: 0, 2024-09-15T04:05:03.933 INFO:teuthology.orchestra.run.smithi110.stdout: 1, 2024-09-15T04:05:03.933 INFO:teuthology.orchestra.run.smithi110.stdout: 2 2024-09-15T04:05:03.933 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.934 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_age": 0, 2024-09-15T04:05:03.934 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:05:03.934 INFO:teuthology.orchestra.run.smithi110.stdout: "required_con": "2449958755906961412", 2024-09-15T04:05:03.934 INFO:teuthology.orchestra.run.smithi110.stdout: "required_mon": [ 2024-09-15T04:05:03.934 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:05:03.934 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:05:03.934 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:05:03.934 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:05:03.934 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:05:03.934 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:05:03.934 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:05:03.934 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:05:03.934 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:05:03.934 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.935 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_con": "4540138320759226367", 2024-09-15T04:05:03.935 INFO:teuthology.orchestra.run.smithi110.stdout: "quorum_mon": [ 2024-09-15T04:05:03.935 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:05:03.935 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:05:03.935 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:05:03.935 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:05:03.935 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:05:03.935 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:05:03.935 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:05:03.935 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:05:03.935 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:05:03.935 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.935 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.935 INFO:teuthology.orchestra.run.smithi110.stdout: "outside_quorum": [], 2024-09-15T04:05:03.936 INFO:teuthology.orchestra.run.smithi110.stdout: "extra_probe_peers": [], 2024-09-15T04:05:03.936 INFO:teuthology.orchestra.run.smithi110.stdout: "sync_provider": [], 2024-09-15T04:05:03.936 INFO:teuthology.orchestra.run.smithi110.stdout: "monmap": { 2024-09-15T04:05:03.936 INFO:teuthology.orchestra.run.smithi110.stdout: "epoch": 1, 2024-09-15T04:05:03.936 INFO:teuthology.orchestra.run.smithi110.stdout: "fsid": "802f2aac-d77f-438d-80e9-5e0c23519086", 2024-09-15T04:05:03.936 INFO:teuthology.orchestra.run.smithi110.stdout: "modified": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:05:03.936 INFO:teuthology.orchestra.run.smithi110.stdout: "created": "2024-09-15T03:58:15.825657Z", 2024-09-15T04:05:03.936 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release": 17, 2024-09-15T04:05:03.936 INFO:teuthology.orchestra.run.smithi110.stdout: "min_mon_release_name": "quincy", 2024-09-15T04:05:03.936 INFO:teuthology.orchestra.run.smithi110.stdout: "election_strategy": 1, 2024-09-15T04:05:03.936 INFO:teuthology.orchestra.run.smithi110.stdout: "disallowed_leaders: ": "", 2024-09-15T04:05:03.936 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false, 2024-09-15T04:05:03.936 INFO:teuthology.orchestra.run.smithi110.stdout: "tiebreaker_mon": "", 2024-09-15T04:05:03.937 INFO:teuthology.orchestra.run.smithi110.stdout: "removed_ranks: ": "", 2024-09-15T04:05:03.937 INFO:teuthology.orchestra.run.smithi110.stdout: "features": { 2024-09-15T04:05:03.937 INFO:teuthology.orchestra.run.smithi110.stdout: "persistent": [ 2024-09-15T04:05:03.937 INFO:teuthology.orchestra.run.smithi110.stdout: "kraken", 2024-09-15T04:05:03.937 INFO:teuthology.orchestra.run.smithi110.stdout: "luminous", 2024-09-15T04:05:03.937 INFO:teuthology.orchestra.run.smithi110.stdout: "mimic", 2024-09-15T04:05:03.937 INFO:teuthology.orchestra.run.smithi110.stdout: "osdmap-prune", 2024-09-15T04:05:03.937 INFO:teuthology.orchestra.run.smithi110.stdout: "nautilus", 2024-09-15T04:05:03.937 INFO:teuthology.orchestra.run.smithi110.stdout: "octopus", 2024-09-15T04:05:03.937 INFO:teuthology.orchestra.run.smithi110.stdout: "pacific", 2024-09-15T04:05:03.937 INFO:teuthology.orchestra.run.smithi110.stdout: "elector-pinging", 2024-09-15T04:05:03.937 INFO:teuthology.orchestra.run.smithi110.stdout: "quincy" 2024-09-15T04:05:03.937 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.937 INFO:teuthology.orchestra.run.smithi110.stdout: "optional": [] 2024-09-15T04:05:03.938 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.938 INFO:teuthology.orchestra.run.smithi110.stdout: "mons": [ 2024-09-15T04:05:03.938 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.938 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 0, 2024-09-15T04:05:03.938 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "a", 2024-09-15T04:05:03.938 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:05:03.938 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:05:03.938 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.938 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:05:03.938 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3300", 2024-09-15T04:05:03.938 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.938 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.938 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.938 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:05:03.939 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789", 2024-09-15T04:05:03.939 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.939 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.939 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.939 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.939 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6789/0", 2024-09-15T04:05:03.939 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6789/0", 2024-09-15T04:05:03.939 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:05:03.939 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:05:03.939 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:05:03.939 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.939 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.939 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 1, 2024-09-15T04:05:03.939 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "b", 2024-09-15T04:05:03.940 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:05:03.940 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:05:03.940 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.940 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:05:03.940 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:3300", 2024-09-15T04:05:03.940 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.940 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.940 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.940 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:05:03.940 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789", 2024-09-15T04:05:03.940 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.940 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.940 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.940 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.941 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.177:6789/0", 2024-09-15T04:05:03.941 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.177:6789/0", 2024-09-15T04:05:03.941 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:05:03.941 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:05:03.941 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:05:03.941 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.941 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.941 INFO:teuthology.orchestra.run.smithi110.stdout: "rank": 2, 2024-09-15T04:05:03.941 INFO:teuthology.orchestra.run.smithi110.stdout: "name": "c", 2024-09-15T04:05:03.941 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addrs": { 2024-09-15T04:05:03.941 INFO:teuthology.orchestra.run.smithi110.stdout: "addrvec": [ 2024-09-15T04:05:03.941 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.941 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v2", 2024-09-15T04:05:03.942 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:3301", 2024-09-15T04:05:03.942 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.942 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.942 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.942 INFO:teuthology.orchestra.run.smithi110.stdout: "type": "v1", 2024-09-15T04:05:03.942 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790", 2024-09-15T04:05:03.942 INFO:teuthology.orchestra.run.smithi110.stdout: "nonce": 0 2024-09-15T04:05:03.942 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.942 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.942 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.942 INFO:teuthology.orchestra.run.smithi110.stdout: "addr": "172.21.15.110:6790/0", 2024-09-15T04:05:03.942 INFO:teuthology.orchestra.run.smithi110.stdout: "public_addr": "172.21.15.110:6790/0", 2024-09-15T04:05:03.942 INFO:teuthology.orchestra.run.smithi110.stdout: "priority": 0, 2024-09-15T04:05:03.942 INFO:teuthology.orchestra.run.smithi110.stdout: "weight": 0, 2024-09-15T04:05:03.943 INFO:teuthology.orchestra.run.smithi110.stdout: "crush_location": "{}" 2024-09-15T04:05:03.943 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.943 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.943 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.943 INFO:teuthology.orchestra.run.smithi110.stdout: "feature_map": { 2024-09-15T04:05:03.943 INFO:teuthology.orchestra.run.smithi110.stdout: "mon": [ 2024-09-15T04:05:03.943 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.943 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:05:03.943 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:05:03.943 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 1 2024-09-15T04:05:03.943 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.943 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.943 INFO:teuthology.orchestra.run.smithi110.stdout: "osd": [ 2024-09-15T04:05:03.944 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.944 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:05:03.944 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:05:03.944 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 4 2024-09-15T04:05:03.944 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.944 INFO:teuthology.orchestra.run.smithi110.stdout: ], 2024-09-15T04:05:03.944 INFO:teuthology.orchestra.run.smithi110.stdout: "client": [ 2024-09-15T04:05:03.944 INFO:teuthology.orchestra.run.smithi110.stdout: { 2024-09-15T04:05:03.944 INFO:teuthology.orchestra.run.smithi110.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T04:05:03.944 INFO:teuthology.orchestra.run.smithi110.stdout: "release": "luminous", 2024-09-15T04:05:03.944 INFO:teuthology.orchestra.run.smithi110.stdout: "num": 5 2024-09-15T04:05:03.944 INFO:teuthology.orchestra.run.smithi110.stdout: } 2024-09-15T04:05:03.944 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-09-15T04:05:03.944 INFO:teuthology.orchestra.run.smithi110.stdout: }, 2024-09-15T04:05:03.945 INFO:teuthology.orchestra.run.smithi110.stdout: "stretch_mode": false 2024-09-15T04:05:03.945 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:05:03.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b9219d640 1 -- 172.21.15.110:0/3356766683 >> v2:172.21.15.177:6812/28201 conn(0x7f7b78038730 msgr2=0x7f7b7803abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b9219d640 1 --2- 172.21.15.110:0/3356766683 >> v2:172.21.15.177:6812/28201 conn(0x7f7b78038730 0x7f7b7803abe0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7b80009130 tx=0x7f7b80019000 comp rx=0 tx=0).stop 2024-09-15T04:05:03.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b9219d640 10 monclient: shutdown 2024-09-15T04:05:03.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b9219d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:03.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b9219d640 1 -- 172.21.15.110:0/3356766683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b8c130d00 msgr2=0x7f7b8c099a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:03.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b9219d640 1 --2- 172.21.15.110:0/3356766683 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7b8c130d00 0x7f7b8c099a90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7b840099f0 tx=0x7f7b84037000 comp rx=0 tx=0).stop 2024-09-15T04:05:03.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b8bfff640 1 -- 172.21.15.110:0/3356766683 reap_dead start 2024-09-15T04:05:03.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b8bfff640 1 -- 172.21.15.110:0/3356766683 reap_dead start 2024-09-15T04:05:03.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b9219d640 1 -- 172.21.15.110:0/3356766683 shutdown_connections 2024-09-15T04:05:03.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b9219d640 1 -- 172.21.15.110:0/3356766683 >> 172.21.15.110:0/3356766683 conn(0x7f7b8c062f70 msgr2=0x7f7b8c09f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:03.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b9219d640 1 -- 172.21.15.110:0/3356766683 shutdown_connections 2024-09-15T04:05:03.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:03.921+0000 7f7b9219d640 1 -- 172.21.15.110:0/3356766683 wait complete. 2024-09-15T04:05:03.947 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T04:05:03.947 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T04:05:03.947 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T04:05:04.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.038+0000 7f89cb220640 10 monclient: get_monmap_and_config 2024-09-15T04:05:04.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.038+0000 7f89cb220640 10 monclient: build_initial_monmap 2024-09-15T04:05:04.039 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.038+0000 7f89cb220640 10 monclient: monmap: 2024-09-15T04:05:04.039 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:04.039 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:04.040 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:04.039795+0000 2024-09-15T04:05:04.040 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:04.039795+0000 2024-09-15T04:05:04.040 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:04.040 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:04.040 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:04.040 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:04.040 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:04.040 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:04.040 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:04.040 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:04.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.039+0000 7f89cb220640 1 Processor -- start 2024-09-15T04:05:04.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.039+0000 7f89cb220640 1 -- start start 2024-09-15T04:05:04.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.039+0000 7f89cb220640 10 monclient: init 2024-09-15T04:05:04.042 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.039+0000 7f89cb220640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:04.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.039+0000 7f89cb220640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-15T04:05:04.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89cb220640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 0x7f89c409ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:04.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89cb220640 10 monclient(hunting): picked mon.noname-b con 0x7f89c406e590 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:04.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89cb220640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f89c409e2f0 0x7f89c412f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:04.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89cb220640 10 monclient(hunting): picked mon.noname-c con 0x7f89c409e2f0 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:04.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89cb220640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89c409e6c0 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:04.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89cb220640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89c409f340 con 0x7f89c409e6c0 2024-09-15T04:05:04.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89cb220640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:04.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89cb220640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89c406dab0 con 0x7f89c409e2f0 2024-09-15T04:05:04.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89cb220640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:04.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89cb220640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89c406e960 con 0x7f89c406e590 2024-09-15T04:05:04.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89cb220640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:04.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89cb220640 10 monclient(hunting): authenticate will time out at 1127.684070s 2024-09-15T04:05:04.043 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 0x7f89c409ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:04.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 0x7f89c409ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35102/0 (socket says 172.21.15.110:35102) 2024-09-15T04:05:04.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c97bd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f89c409e6c0 0x7f89c412f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58914/0 (socket says 172.21.15.110:58914) 2024-09-15T04:05:04.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 1 -- 172.21.15.110:0/638218269 learned_addr learned my addr 172.21.15.110:0/638218269 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:04.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 10 monclient(hunting): get_auth_request con 0x7f89c406e590 auth_method 0 2024-09-15T04:05:04.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:04.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:04.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:04.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:04.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:04.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:04.044 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 10 monclient(hunting): handle_auth_done global_id 74123 payload 274 2024-09-15T04:05:04.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 1 -- 172.21.15.110:0/638218269 >> v2:172.21.15.177:3300/0 conn(0x7f89c409e2f0 msgr2=0x7f89c412f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:04.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89bbfff640 1 --2- 172.21.15.110:0/638218269 >> v2:172.21.15.177:3300/0 conn(0x7f89c409e2f0 0x7f89c412f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:04.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 1 --2- 172.21.15.110:0/638218269 >> v2:172.21.15.177:3300/0 conn(0x7f89c409e2f0 0x7f89c412f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:04.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 1 -- 172.21.15.110:0/638218269 >> v1:172.21.15.110:6789/0 conn(0x7f89c409e6c0 legacy=0x7f89c412f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:04.045 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 10 monclient: _finish_hunting 0 2024-09-15T04:05:04.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 1 monclient: found mon.noname-b 2024-09-15T04:05:04.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:04.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:05:04.046 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.040+0000 7f89c8fbc640 1 -- 172.21.15.110:0/638218269 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c406f040 con 0x7f89c406e590 2024-09-15T04:05:04.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89c8fbc640 1 -- 172.21.15.110:0/638218269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 msgr2=0x7f89c409ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:04.051 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89c8fbc640 1 --2- 172.21.15.110:0/638218269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 0x7f89c409ddb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009940 tx=0x7f89c00091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47cdb0aa82dda109 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:04.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89bb7fe640 1 -- 172.21.15.110:0/638218269 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c0032440 con 0x7f89c406e590 2024-09-15T04:05:04.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:04.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:05:04.052 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89bb7fe640 10 monclient: dump: 2024-09-15T04:05:04.052 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:04.052 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:04.052 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:04.052 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:04.052 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:04.052 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:04.052 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:04.052 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:04.052 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89bb7fe640 10 monclient: _finish_auth 0 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89bb7fe640 1 -- 172.21.15.110:0/638218269 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c00325a0 con 0x7f89c406e590 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89cb220640 5 monclient: authenticate success, global_id 74123 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89cb220640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89cb220640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89bb7fe640 1 -- 172.21.15.110:0/638218269 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c0032940 con 0x7f89c406e590 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89cb220640 4 set_mon_vals no callback set 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89bb7fe640 10 monclient: dump: 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:04.053 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:04.054 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:04.054 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:04.054 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:04.054 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:04.054 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:04.054 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:04.054 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:04.054 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:04.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89cb220640 10 monclient: shutdown 2024-09-15T04:05:04.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89cb220640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:04.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89cb220640 1 -- 172.21.15.110:0/638218269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 msgr2=0x7f89c409ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:04.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89cb220640 1 --2- 172.21.15.110:0/638218269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 0x7f89c409ddb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009940 tx=0x7f89c00091d0 comp rx=0 tx=0).stop 2024-09-15T04:05:04.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89cb220640 1 -- 172.21.15.110:0/638218269 shutdown_connections 2024-09-15T04:05:04.054 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89cb220640 1 --2- 172.21.15.110:0/638218269 >> v2:172.21.15.177:3300/0 conn(0x7f89c409e2f0 0x7f89c412f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:04.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89cb220640 1 --2- 172.21.15.110:0/638218269 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 0x7f89c409ddb0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:04.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89cb220640 1 -- 172.21.15.110:0/638218269 >> 172.21.15.110:0/638218269 conn(0x7f89c40639f0 msgr2=0x7f89c40a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:04.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89cb220640 1 -- 172.21.15.110:0/638218269 shutdown_connections 2024-09-15T04:05:04.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.043+0000 7f89cb220640 1 -- 172.21.15.110:0/638218269 wait complete. 2024-09-15T04:05:04.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.044+0000 7f89cb220640 10 monclient: build_initial_monmap 2024-09-15T04:05:04.055 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.044+0000 7f89cb220640 10 monclient: monmap: 2024-09-15T04:05:04.055 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:04.055 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:04.055 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:04.055 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:04.055 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:04.055 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:04.055 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:04.055 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.044+0000 7f89cb220640 1 Processor -- start 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.044+0000 7f89cb220640 1 -- start start 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.044+0000 7f89cb220640 10 monclient: init 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 0x7f89c413ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 10 monclient(hunting): picked mon.noname-c con 0x7f89c406e590 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89c409e2f0 0x7f89c412f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 10 monclient(hunting): picked mon.noname-b con 0x7f89c409e2f0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89c409e6c0 0x7f89c41191c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 10 monclient(hunting): picked mon.noname-a con 0x7f89c409e6c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:04.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c411ac80 con 0x7f89c409e6c0 2024-09-15T04:05:04.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:04.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89c409cdd0 con 0x7f89c409e2f0 2024-09-15T04:05:04.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:04.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89c411c680 con 0x7f89c406e590 2024-09-15T04:05:04.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:04.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89cb220640 10 monclient(hunting): authenticate will time out at 1127.689317s 2024-09-15T04:05:04.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 0x7f89c413ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:04.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c97bd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89c409e6c0 0x7f89c41191c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:04.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 0x7f89c413ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35110/0 (socket says 172.21.15.110:35110) 2024-09-15T04:05:04.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 1 -- 172.21.15.110:0/4247837527 learned_addr learned my addr 172.21.15.110:0/4247837527 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:04.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 10 monclient(hunting): get_auth_request con 0x7f89c406e590 auth_method 0 2024-09-15T04:05:04.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:04.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:04.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:04.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c97bd640 10 monclient(hunting): get_auth_request con 0x7f89c409e6c0 auth_method 0 2024-09-15T04:05:04.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c97bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:04.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c97bd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:04.058 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c97bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:04.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:04.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:04.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:04.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c97bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:04.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c97bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:04.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c97bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:04.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89bbfff640 1 --2- 172.21.15.110:0/4247837527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89c409e2f0 0x7f89c412f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:04.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 10 monclient(hunting): handle_auth_done global_id 74126 payload 824 2024-09-15T04:05:04.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 1 -- 172.21.15.110:0/4247837527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89c409e2f0 msgr2=0x7f89c412f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:04.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 1 --2- 172.21.15.110:0/4247837527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89c409e2f0 0x7f89c412f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:04.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 1 -- 172.21.15.110:0/4247837527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89c409e6c0 msgr2=0x7f89c41191c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:04.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 1 --2- 172.21.15.110:0/4247837527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89c409e6c0 0x7f89c41191c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:04.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 10 monclient: _finish_hunting 0 2024-09-15T04:05:04.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 1 monclient: found mon.noname-c 2024-09-15T04:05:04.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:04.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:04.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c8fbc640 1 -- 172.21.15.110:0/4247837527 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c0009450 con 0x7f89c406e590 2024-09-15T04:05:04.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89bbfff640 10 monclient: get_auth_request con 0x7f89c409e2f0 auth_method 0 2024-09-15T04:05:04.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89c97bd640 1 --2- 172.21.15.110:0/4247837527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89c409e6c0 0x7f89c41191c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:04.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.045+0000 7f89bbfff640 1 --2- 172.21.15.110:0/4247837527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89c409e2f0 0x7f89c412f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:05:04.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.047+0000 7f89c8fbc640 1 -- 172.21.15.110:0/4247837527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 msgr2=0x7f89c413ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:04.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.047+0000 7f89c8fbc640 1 --2- 172.21.15.110:0/4247837527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 0x7f89c413ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f89c0035040 tx=0x7f89c0032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:04.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89b8ff9640 1 -- 172.21.15.110:0/4247837527 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c0031a10 con 0x7f89c406e590 2024-09-15T04:05:04.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:04.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89b8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:04.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89b8ff9640 10 monclient: dump: 2024-09-15T04:05:04.060 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:04.060 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:04.061 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:04.061 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:04.061 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:04.061 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:04.061 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:04.061 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:04.061 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:04.061 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:04.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89b8ff9640 10 monclient: _finish_auth 0 2024-09-15T04:05:04.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:04.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89b8ff9640 1 -- 172.21.15.110:0/4247837527 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c0031b90 con 0x7f89c406e590 2024-09-15T04:05:04.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89cb220640 5 monclient: authenticate success, global_id 74126 2024-09-15T04:05:04.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:04.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89b8ff9640 1 -- 172.21.15.110:0/4247837527 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c003b400 con 0x7f89c406e590 2024-09-15T04:05:04.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:04.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89b8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:04.062 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89b8ff9640 10 monclient: dump: 2024-09-15T04:05:04.062 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:04.062 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:04.062 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:04.062 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:04.062 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:04.062 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:04.062 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:04.063 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:04.063 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:04.063 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:04.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89cb220640 10 monclient: _renew_subs 2024-09-15T04:05:04.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89cb220640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:04.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89cb220640 1 -- 172.21.15.110:0/4247837527 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89c411c960 con 0x7f89c406e590 2024-09-15T04:05:04.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89baffd640 4 set_mon_vals no callback set 2024-09-15T04:05:04.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89cb220640 10 monclient: _renew_subs 2024-09-15T04:05:04.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89cb220640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:04.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.048+0000 7f89cb220640 1 -- 172.21.15.110:0/4247837527 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89c413f140 con 0x7f89c406e590 2024-09-15T04:05:04.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.049+0000 7f89b8ff9640 1 -- 172.21.15.110:0/4247837527 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f89c0039040 con 0x7f89c406e590 2024-09-15T04:05:04.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.049+0000 7f89cb220640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:04.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.049+0000 7f89cb220640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:04.063 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.049+0000 7f89cb220640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:04.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.049+0000 7f89cb220640 1 -- 172.21.15.110:0/4247837527 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89c411cdf0 con 0x7f89c406e590 2024-09-15T04:05:04.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.049+0000 7f89b8ff9640 1 --2- 172.21.15.110:0/4247837527 >> v2:172.21.15.177:6812/28201 conn(0x7f89b0038700 0x7f89b003abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:04.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.049+0000 7f89b8ff9640 1 -- 172.21.15.110:0/4247837527 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f89c006ba40 con 0x7f89c406e590 2024-09-15T04:05:04.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.049+0000 7f89bbfff640 1 --2- 172.21.15.110:0/4247837527 >> v2:172.21.15.177:6812/28201 conn(0x7f89b0038700 0x7f89b003abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:04.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.049+0000 7f89bbfff640 10 monclient: get_auth_request con 0x7f89b0038700 auth_method 0 2024-09-15T04:05:04.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.052+0000 7f89bbfff640 1 --2- 172.21.15.110:0/4247837527 >> v2:172.21.15.177:6812/28201 conn(0x7f89b0038700 0x7f89b003abb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f89b4008960 tx=0x7f89b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:04.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.053+0000 7f89b8ff9640 1 -- 172.21.15.110:0/4247837527 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f89c411cdf0 con 0x7f89c406e590 2024-09-15T04:05:04.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.053+0000 7f89b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:04.065 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.053+0000 7f89b8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:04.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.269+0000 7f89cb220640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T04:05:04.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.269+0000 7f89cb220640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T04:05:04.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.269+0000 7f89cb220640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:04.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.269+0000 7f89cb220640 1 -- 172.21.15.110:0/4247837527 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f89c411cdd0 con 0x7f89c406e590 2024-09-15T04:05:04.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.270+0000 7f89b8ff9640 1 -- 172.21.15.110:0/4247837527 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (secure 0 0 0) 0x7f89c0037030 con 0x7f89c406e590 2024-09-15T04:05:04.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.270+0000 7f89b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T04:05:04.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.270+0000 7f89b8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:05:04.271 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.270+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:04.271 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.270+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=7,logm=65} crc {auth=435840844,config=3140514009,health=1017346500,kv=398645650,logm=1855516433}) 2024-09-15T04:05:04.271 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.270+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=12,kv=7,logm=66} crc {auth=2513199409,config=3140514009,health=1017346500,kv=2345446280,logm=1502059825}) 2024-09-15T04:05:04.272 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.270+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:04.272 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.270+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=7,logm=65} crc {auth=435840844,config=3140514009,health=1017346500,kv=398645650,logm=1855516433}) 2024-09-15T04:05:04.272 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.270+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=11,kv=8,logm=65} crc {auth=435840844,config=3140514009,health=3625402836,kv=1242700903,logm=3647090355}) 2024-09-15T04:05:04.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.272+0000 7f89cb220640 1 -- 172.21.15.110:0/4247837527 >> v2:172.21.15.177:6812/28201 conn(0x7f89b0038700 msgr2=0x7f89b003abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:04.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.272+0000 7f89cb220640 1 --2- 172.21.15.110:0/4247837527 >> v2:172.21.15.177:6812/28201 conn(0x7f89b0038700 0x7f89b003abb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f89b4008960 tx=0x7f89b4008040 comp rx=0 tx=0).stop 2024-09-15T04:05:04.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.272+0000 7f89cb220640 10 monclient: shutdown 2024-09-15T04:05:04.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.272+0000 7f89cb220640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:04.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.272+0000 7f89cb220640 1 -- 172.21.15.110:0/4247837527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 msgr2=0x7f89c413ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:04.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.272+0000 7f89cb220640 1 --2- 172.21.15.110:0/4247837527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 0x7f89c413ec00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f89c0035040 tx=0x7f89c0032000 comp rx=0 tx=0).stop 2024-09-15T04:05:04.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.272+0000 7f89cb220640 1 -- 172.21.15.110:0/4247837527 shutdown_connections 2024-09-15T04:05:04.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.272+0000 7f89cb220640 1 --2- 172.21.15.110:0/4247837527 >> v2:172.21.15.177:6812/28201 conn(0x7f89b0038700 0x7f89b003abb0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:04.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.272+0000 7f89cb220640 1 --2- 172.21.15.110:0/4247837527 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f89c409e6c0 0x7f89c41191c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:04.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.272+0000 7f89cb220640 1 --2- 172.21.15.110:0/4247837527 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89c409e2f0 0x7f89c412f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:04.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.272+0000 7f89cb220640 1 --2- 172.21.15.110:0/4247837527 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f89c406e590 0x7f89c413ec00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:04.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.272+0000 7f89cb220640 1 -- 172.21.15.110:0/4247837527 >> 172.21.15.110:0/4247837527 conn(0x7f89c40639f0 msgr2=0x7f89c40a17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:04.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.273+0000 7f89cb220640 1 -- 172.21.15.110:0/4247837527 shutdown_connections 2024-09-15T04:05:04.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:04.273+0000 7f89cb220640 1 -- 172.21.15.110:0/4247837527 wait complete. 2024-09-15T04:05:04.283 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T04:05:04.937 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.937+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:04.938 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.937+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2557406174}) 2024-09-15T04:05:04.938 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.937+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1430236294}) 2024-09-15T04:05:04.938 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.937+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:04.938 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.937+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2557406174}) 2024-09-15T04:05:04.938 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.937+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2738714290}) 2024-09-15T04:05:04.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.938+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:04.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.938+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3264557557}) 2024-09-15T04:05:04.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.938+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3892929644}) 2024-09-15T04:05:04.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.938+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:04.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.938+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3264557557}) 2024-09-15T04:05:04.939 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.938+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389288421}) 2024-09-15T04:05:04.940 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.939+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:04.940 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.939+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521779173}) 2024-09-15T04:05:04.940 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.939+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=584925083}) 2024-09-15T04:05:04.940 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.939+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:04.940 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.939+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521779173}) 2024-09-15T04:05:04.940 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.939+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2162244008}) 2024-09-15T04:05:04.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.940+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:04.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.940+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=810976476}) 2024-09-15T04:05:04.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.940+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2537453670}) 2024-09-15T04:05:04.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.940+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:04.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.940+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=810976476}) 2024-09-15T04:05:04.941 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:04.940+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=8464041}) 2024-09-15T04:05:05.163 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.162+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.163 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.162+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2535027908}) 2024-09-15T04:05:05.163 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.162+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4188602902}) 2024-09-15T04:05:05.163 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.162+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.163 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.162+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2535027908}) 2024-09-15T04:05:05.163 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.162+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2079835443}) 2024-09-15T04:05:05.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.227+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.227+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1627029564}) 2024-09-15T04:05:05.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.227+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=982768405}) 2024-09-15T04:05:05.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.227+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.227+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1627029564}) 2024-09-15T04:05:05.228 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.227+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1752650216}) 2024-09-15T04:05:05.231 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.230+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.231 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.230+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1935825135}) 2024-09-15T04:05:05.231 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.230+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=677332520}) 2024-09-15T04:05:05.231 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.230+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.231 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.230+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1935825135}) 2024-09-15T04:05:05.231 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.230+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1993684849}) 2024-09-15T04:05:05.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.235+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.235+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783510728}) 2024-09-15T04:05:05.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.235+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1118146489}) 2024-09-15T04:05:05.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.235+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.235+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783510728}) 2024-09-15T04:05:05.236 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.235+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1373366616}) 2024-09-15T04:05:05.284 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T04:05:05.284 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T04:05:05.384 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.383+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.384 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.383+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436345827}) 2024-09-15T04:05:05.384 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.383+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2394962157}) 2024-09-15T04:05:05.384 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.383+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.384 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.383+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436345827}) 2024-09-15T04:05:05.384 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.383+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1615013026}) 2024-09-15T04:05:05.386 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.384+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.386 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.384+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3771379059}) 2024-09-15T04:05:05.386 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.384+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1044795587}) 2024-09-15T04:05:05.386 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.384+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.384+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3771379059}) 2024-09-15T04:05:05.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.384+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2618009706}) 2024-09-15T04:05:05.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.384+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.385+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4258247637}) 2024-09-15T04:05:05.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.385+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1648398292}) 2024-09-15T04:05:05.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.385+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.385+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4258247637}) 2024-09-15T04:05:05.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.385+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1537312021}) 2024-09-15T04:05:05.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.385+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.385+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1066849328}) 2024-09-15T04:05:05.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.385+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1130985269}) 2024-09-15T04:05:05.387 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.385+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.388 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.385+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1066849328}) 2024-09-15T04:05:05.388 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.385+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1655439203}) 2024-09-15T04:05:05.388 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.386+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.388 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.386+0000 7f028f2c0640 10 monclient: get_monmap_and_config 2024-09-15T04:05:05.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.386+0000 7f028f2c0640 10 monclient: build_initial_monmap 2024-09-15T04:05:05.389 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.386+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=4047812223,mdsmap=1531151339,mgr=3027731993}) 2024-09-15T04:05:05.389 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.386+0000 7f028f2c0640 10 monclient: monmap: 2024-09-15T04:05:05.389 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:05.389 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:05.389 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:05.387555+0000 2024-09-15T04:05:05.389 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:05.387555+0000 2024-09-15T04:05:05.390 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:05.390 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:05.390 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:05.390 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:05.390 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:05.390 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:05.390 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:05.390 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:05.391 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.386+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=3444634005,mdsmap=1531151339,mgr=1004108257}) 2024-09-15T04:05:05.391 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.386+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.391 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.386+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=4047812223,mdsmap=1531151339,mgr=3027731993}) 2024-09-15T04:05:05.391 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.386+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=2671732449,mdsmap=1531151339,mgr=1396177949,mgr_command_descs=3545737492,mgr_metadata=3491211682,mgrstat=1567338541}) 2024-09-15T04:05:05.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f028f2c0640 1 Processor -- start 2024-09-15T04:05:05.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f028f2c0640 1 -- start start 2024-09-15T04:05:05.391 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f028f2c0640 10 monclient: init 2024-09-15T04:05:05.392 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.393 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=67} crc {mgr=543305786,mgr_command_descs=3545737492,mgr_metadata=3491211682,mgrstat=1624772194,mon_config_key=3532925585,monmap=2614306414,osd_metadata=3281624269,osd_pg_creating=3286416672,osd_snap=3178633393,osdmap=878660713}) 2024-09-15T04:05:05.393 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=6,osd_snap=2,osdmap=69} crc {mgr=2586702516,mgr_command_descs=3545737492,mgrstat=1877648920,mon_config_key=3532925585,monmap=3538901833,osd_metadata=3281624269,osd_snap=3178633393,osdmap=2313824938}) 2024-09-15T04:05:05.393 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.393 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=67} crc {mgr=543305786,mgr_command_descs=3545737492,mgr_metadata=3491211682,mgrstat=1624772194,mon_config_key=3532925585,monmap=2614306414,osd_metadata=3281624269,osd_pg_creating=3286416672,osd_snap=3178633393,osdmap=878660713}) 2024-09-15T04:05:05.393 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=3,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=1,osdmap=68} crc {mgr=543305786,mgr_command_descs=3545737492,mgr_metadata=3491211682,mgrstat=3167387796,mon_config_key=3532925585,monmap=3538901833,osd_metadata=3281624269,osd_pg_creating=3286416672,osd_snap=956710772,osdmap=1404902875}) 2024-09-15T04:05:05.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f028f2c0640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:05.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f028f2c0640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-15T04:05:05.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f028f2c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02880a1a60 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:05.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f028f2c0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 0x7f0288131a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:05.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f028f2c0640 10 monclient(hunting): picked mon.noname-b con 0x7f0288131640 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:05.395 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.387+0000 7f028f2c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0288131f50 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:05.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028f2c0640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02880a1260 con 0x7f02880a1a60 2024-09-15T04:05:05.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028f2c0640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f028809c870 con 0x7f0288131f50 2024-09-15T04:05:05.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028f2c0640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:05.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028f2c0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02880a0c60 con 0x7f0288131640 2024-09-15T04:05:05.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028f2c0640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:05.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028f2c0640 10 monclient(hunting): authenticate will time out at 1129.032045s 2024-09-15T04:05:05.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028d05c640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f02880a1a60 0x7f028809d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58928/0 (socket says 172.21.15.110:58928) 2024-09-15T04:05:05.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028d05c640 1 -- 172.21.15.110:0/48596816 learned_addr learned my addr 172.21.15.110:0/48596816 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:05.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 1 --2- 172.21.15.110:0/48596816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 0x7f0288131a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:05.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 10 monclient(hunting): get_auth_request con 0x7f0288131640 auth_method 0 2024-09-15T04:05:05.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:05.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:05.396 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 10 monclient(hunting): handle_auth_done global_id 74129 payload 274 2024-09-15T04:05:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 1 -- 172.21.15.110:0/48596816 >> v1:172.21.15.177:6789/0 conn(0x7f0288131f50 legacy=0x7f0288138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 1 -- 172.21.15.110:0/48596816 >> v1:172.21.15.110:6789/0 conn(0x7f02880a1a60 legacy=0x7f028809d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 10 monclient: _finish_hunting 0 2024-09-15T04:05:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 1 monclient: found mon.noname-b 2024-09-15T04:05:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:05:05.397 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.388+0000 7f028c85b640 1 -- 172.21.15.110:0/48596816 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028809cb10 con 0x7f0288131640 2024-09-15T04:05:05.398 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.390+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.398 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.390+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3478985155}) 2024-09-15T04:05:05.398 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.390+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2372235786}) 2024-09-15T04:05:05.398 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.390+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.399 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.390+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3478985155}) 2024-09-15T04:05:05.399 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.390+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3366578075}) 2024-09-15T04:05:05.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f028c85b640 1 -- 172.21.15.110:0/48596816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 msgr2=0x7f0288131a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:05.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f028c85b640 1 --2- 172.21.15.110:0/48596816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 0x7f0288131a10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0278009880 tx=0x7f02780090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17eefe613741c6e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:05.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f027ffff640 1 -- 172.21.15.110:0/48596816 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278032440 con 0x7f0288131640 2024-09-15T04:05:05.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f027ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:05.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3679199510}) 2024-09-15T04:05:05.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3918092957}) 2024-09-15T04:05:05.400 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f027ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:05:05.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.400 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3679199510}) 2024-09-15T04:05:05.401 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3364855644}) 2024-09-15T04:05:05.402 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f027ffff640 10 monclient: dump: 2024-09-15T04:05:05.402 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:05.402 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:05.402 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.402 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.402 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:05.402 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:05.402 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:05.402 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:05.402 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:05.403 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f027ffff640 10 monclient: _finish_auth 0 2024-09-15T04:05:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f027ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f027ffff640 1 -- 172.21.15.110:0/48596816 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02780325a0 con 0x7f0288131640 2024-09-15T04:05:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f028f2c0640 5 monclient: authenticate success, global_id 74129 2024-09-15T04:05:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f028f2c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f027ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f028f2c0640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f028f2c0640 4 set_mon_vals no callback set 2024-09-15T04:05:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f027ffff640 1 -- 172.21.15.110:0/48596816 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278032940 con 0x7f0288131640 2024-09-15T04:05:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f027ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f027ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.391+0000 7f027ffff640 10 monclient: dump: 2024-09-15T04:05:05.403 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:05.404 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:05.404 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.404 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.404 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:05.404 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:05.404 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:05.404 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:05.404 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:05.404 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:05.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f028f2c0640 10 monclient: shutdown 2024-09-15T04:05:05.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f028f2c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:05.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f028f2c0640 1 -- 172.21.15.110:0/48596816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 msgr2=0x7f0288131a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:05.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f028f2c0640 1 --2- 172.21.15.110:0/48596816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 0x7f0288131a10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0278009880 tx=0x7f02780090f0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f028f2c0640 1 -- 172.21.15.110:0/48596816 shutdown_connections 2024-09-15T04:05:05.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f028f2c0640 1 --2- 172.21.15.110:0/48596816 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 0x7f0288131a10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f028f2c0640 1 -- 172.21.15.110:0/48596816 >> 172.21.15.110:0/48596816 conn(0x7f0288063270 msgr2=0x7f0288063670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:05.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f028f2c0640 1 -- 172.21.15.110:0/48596816 shutdown_connections 2024-09-15T04:05:05.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f028f2c0640 1 -- 172.21.15.110:0/48596816 wait complete. 2024-09-15T04:05:05.406 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.406 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2885518948}) 2024-09-15T04:05:05.406 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2860266833}) 2024-09-15T04:05:05.406 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.406 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2885518948}) 2024-09-15T04:05:05.406 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3779112971}) 2024-09-15T04:05:05.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f028f2c0640 10 monclient: build_initial_monmap 2024-09-15T04:05:05.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.392+0000 7f028f2c0640 10 monclient: monmap: 2024-09-15T04:05:05.407 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:05.407 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:05.407 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:05.407 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:05.407 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:05.407 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:05.408 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:05.408 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:05.408 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:05.408 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:05.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f028f2c0640 1 Processor -- start 2024-09-15T04:05:05.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f028f2c0640 1 -- start start 2024-09-15T04:05:05.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f028f2c0640 10 monclient: init 2024-09-15T04:05:05.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=921883691}) 2024-09-15T04:05:05.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2088678613}) 2024-09-15T04:05:05.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T04:05:05.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=921883691}) 2024-09-15T04:05:05.409 INFO:tasks.ceph.mon.a.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f91f33f1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3524812595}) 2024-09-15T04:05:05.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f028f2c0640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:05.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f028f2c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T04:05:05.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f028f2c0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02880a1a60 0x7f0288130fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:05.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f028f2c0640 10 monclient(hunting): picked mon.noname-b con 0x7f02880a1a60 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:05.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f028f2c0640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 0x7f0288097160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:05.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f028f2c0640 10 monclient(hunting): picked mon.noname-c con 0x7f0288131640 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:05.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f028f2c0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0288131f50 0x7f0288095a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:05.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f028f2c0640 10 monclient(hunting): picked mon.noname-a con 0x7f0288131f50 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:05.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f028f2c0640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:05.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.393+0000 7f028f2c0640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0288098a50 con 0x7f0288131f50 2024-09-15T04:05:05.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028f2c0640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:05.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028f2c0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0288098d30 con 0x7f02880a1a60 2024-09-15T04:05:05.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028f2c0640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:05.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028f2c0640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0288099010 con 0x7f0288131640 2024-09-15T04:05:05.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028f2c0640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:05.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028f2c0640 10 monclient(hunting): authenticate will time out at 1129.038062s 2024-09-15T04:05:05.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 0x7f0288097160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:05.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d85d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0288131f50 0x7f0288095a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 0x7f0288097160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35134/0 (socket says 172.21.15.110:35134) 2024-09-15T04:05:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 1 -- 172.21.15.110:0/2170858679 learned_addr learned my addr 172.21.15.110:0/2170858679 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 10 monclient(hunting): get_auth_request con 0x7f0288131640 auth_method 0 2024-09-15T04:05:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d85d640 10 monclient(hunting): get_auth_request con 0x7f0288131f50 auth_method 0 2024-09-15T04:05:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d85d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d85d640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d85d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d05c640 1 --2- 172.21.15.110:0/2170858679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02880a1a60 0x7f0288130fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:05.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:05.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:05.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d85d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:05.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d85d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:05.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d85d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:05.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d05c640 10 monclient(hunting): get_auth_request con 0x7f02880a1a60 auth_method 0 2024-09-15T04:05:05.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d05c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:05.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d05c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:05.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d05c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:05.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d05c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:05.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d05c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:05.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d05c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:05.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 10 monclient(hunting): handle_auth_done global_id 74132 payload 824 2024-09-15T04:05:05.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 1 -- 172.21.15.110:0/2170858679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02880a1a60 msgr2=0x7f0288130fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 1 --2- 172.21.15.110:0/2170858679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02880a1a60 0x7f0288130fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 1 -- 172.21.15.110:0/2170858679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0288131f50 msgr2=0x7f0288095a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 1 --2- 172.21.15.110:0/2170858679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0288131f50 0x7f0288095a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 10 monclient: _finish_hunting 0 2024-09-15T04:05:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 1 monclient: found mon.noname-c 2024-09-15T04:05:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028c85b640 1 -- 172.21.15.110:0/2170858679 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0278009390 con 0x7f0288131640 2024-09-15T04:05:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.394+0000 7f028d85d640 1 --2- 172.21.15.110:0/2170858679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0288131f50 0x7f0288095a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f028c85b640 1 -- 172.21.15.110:0/2170858679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 msgr2=0x7f0288097160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f028c85b640 1 --2- 172.21.15.110:0/2170858679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 0x7f0288097160 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0278009360 tx=0x7f0278032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f027d7fa640 1 -- 172.21.15.110:0/2170858679 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278031910 con 0x7f0288131640 2024-09-15T04:05:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f027d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:05.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f027d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:05.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f027d7fa640 10 monclient: dump: 2024-09-15T04:05:05.418 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:05.418 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:05.418 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.418 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.418 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:05.418 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:05.419 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:05.419 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:05.419 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:05.419 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:05.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f027d7fa640 10 monclient: _finish_auth 0 2024-09-15T04:05:05.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f027d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:05.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f027d7fa640 1 -- 172.21.15.110:0/2170858679 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0278031a70 con 0x7f0288131640 2024-09-15T04:05:05.419 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f028f2c0640 5 monclient: authenticate success, global_id 74132 2024-09-15T04:05:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f028f2c0640 10 monclient: _renew_subs 2024-09-15T04:05:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f028f2c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f028f2c0640 1 -- 172.21.15.110:0/2170858679 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f028809aaf0 con 0x7f0288131640 2024-09-15T04:05:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f027d7fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f027d7fa640 1 -- 172.21.15.110:0/2170858679 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0278031e10 con 0x7f0288131640 2024-09-15T04:05:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f027d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f027d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f027d7fa640 10 monclient: dump: 2024-09-15T04:05:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.420 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:05.421 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:05.421 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:05.421 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:05.421 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:05.422 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:05.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.396+0000 7f027f7fe640 4 set_mon_vals no callback set 2024-09-15T04:05:05.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.397+0000 7f028f2c0640 10 monclient: _renew_subs 2024-09-15T04:05:05.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.397+0000 7f028f2c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:05.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.397+0000 7f028f2c0640 1 -- 172.21.15.110:0/2170858679 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0288099500 con 0x7f0288131640 2024-09-15T04:05:05.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.397+0000 7f027d7fa640 1 -- 172.21.15.110:0/2170858679 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f0278039040 con 0x7f0288131640 2024-09-15T04:05:05.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.397+0000 7f028f2c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:05.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.397+0000 7f028f2c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:05.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.397+0000 7f028f2c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:05.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.397+0000 7f028f2c0640 1 -- 172.21.15.110:0/2170858679 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0288070950 con 0x7f0288131640 2024-09-15T04:05:05.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.398+0000 7f027d7fa640 1 --2- 172.21.15.110:0/2170858679 >> v2:172.21.15.177:6812/28201 conn(0x7f026c038700 0x7f026c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:05.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.398+0000 7f027d7fa640 1 -- 172.21.15.110:0/2170858679 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f027806b8c0 con 0x7f0288131640 2024-09-15T04:05:05.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.398+0000 7f028d05c640 1 --2- 172.21.15.110:0/2170858679 >> v2:172.21.15.177:6812/28201 conn(0x7f026c038700 0x7f026c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:05.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.398+0000 7f028d05c640 10 monclient: get_auth_request con 0x7f026c038700 auth_method 0 2024-09-15T04:05:05.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.400+0000 7f028d05c640 1 --2- 172.21.15.110:0/2170858679 >> v2:172.21.15.177:6812/28201 conn(0x7f026c038700 0x7f026c03abb0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0284007fd0 tx=0x7f0284007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:05.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.402+0000 7f027d7fa640 1 -- 172.21.15.110:0/2170858679 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f027803b610 con 0x7f0288131640 2024-09-15T04:05:05.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.402+0000 7f027d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:05.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.402+0000 7f027d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:05.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.615+0000 7f028f2c0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T04:05:05.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.615+0000 7f028f2c0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T04:05:05.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.615+0000 7f028f2c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:05.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.615+0000 7f028f2c0640 1 -- 172.21.15.110:0/2170858679 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0288070950 con 0x7f0288131640 2024-09-15T04:05:05.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.615+0000 7f027d7fa640 1 -- 172.21.15.110:0/2170858679 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1539 (secure 0 0 0) 0x7f027803f020 con 0x7f0288131640 2024-09-15T04:05:05.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.615+0000 7f027d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T04:05:05.616 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.615+0000 7f027d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:05:05.616 INFO:teuthology.orchestra.run.smithi110.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","modified":"2024-09-15T03:58:15.825657Z","created":"2024-09-15T03:58:15.825657Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3301","nonce":0},{"type":"v1","addr":"172.21.15.110:6790","nonce":0}]},"addr":"172.21.15.110:6790/0","public_addr":"172.21.15.110:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T04:05:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.617+0000 7f028f2c0640 1 -- 172.21.15.110:0/2170858679 >> v2:172.21.15.177:6812/28201 conn(0x7f026c038700 msgr2=0x7f026c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.617+0000 7f028f2c0640 1 --2- 172.21.15.110:0/2170858679 >> v2:172.21.15.177:6812/28201 conn(0x7f026c038700 0x7f026c03abb0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0284007fd0 tx=0x7f0284007f00 comp rx=0 tx=0).stop 2024-09-15T04:05:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.617+0000 7f028f2c0640 10 monclient: shutdown 2024-09-15T04:05:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.617+0000 7f028f2c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.617+0000 7f028f2c0640 1 -- 172.21.15.110:0/2170858679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 msgr2=0x7f0288097160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.617+0000 7f028f2c0640 1 --2- 172.21.15.110:0/2170858679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 0x7f0288097160 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0278009360 tx=0x7f0278032000 comp rx=0 tx=0).stop 2024-09-15T04:05:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.617+0000 7f028f2c0640 1 -- 172.21.15.110:0/2170858679 shutdown_connections 2024-09-15T04:05:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.617+0000 7f028f2c0640 1 --2- 172.21.15.110:0/2170858679 >> v2:172.21.15.177:6812/28201 conn(0x7f026c038700 0x7f026c03abb0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.618 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.617+0000 7f028f2c0640 1 --2- 172.21.15.110:0/2170858679 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0288131f50 0x7f0288095a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.617+0000 7f028f2c0640 1 --2- 172.21.15.110:0/2170858679 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f0288131640 0x7f0288097160 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.617+0000 7f028f2c0640 1 --2- 172.21.15.110:0/2170858679 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f02880a1a60 0x7f0288130fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.617+0000 7f028f2c0640 1 -- 172.21.15.110:0/2170858679 >> 172.21.15.110:0/2170858679 conn(0x7f0288063270 msgr2=0x7f028809dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.617+0000 7f028f2c0640 1 -- 172.21.15.110:0/2170858679 shutdown_connections 2024-09-15T04:05:05.619 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.617+0000 7f028f2c0640 1 -- 172.21.15.110:0/2170858679 wait complete. 2024-09-15T04:05:05.628 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T04:05:05.628 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-15T04:05:05.637 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-15T04:05:05.637 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T04:05:05.729 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.729+0000 7fade546f640 10 monclient: get_monmap_and_config 2024-09-15T04:05:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.729+0000 7fade546f640 10 monclient: build_initial_monmap 2024-09-15T04:05:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.729+0000 7fade546f640 10 monclient: monmap: 2024-09-15T04:05:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:05.730129+0000 2024-09-15T04:05:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:05.730129+0000 2024-09-15T04:05:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:05.730 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:05.731 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:05.731 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:05.731 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.729+0000 7fade546f640 1 Processor -- start 2024-09-15T04:05:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.729+0000 7fade546f640 1 -- start start 2024-09-15T04:05:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.729+0000 7fade546f640 10 monclient: init 2024-09-15T04:05:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7fade546f640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7fade546f640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-15T04:05:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7fade546f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fade00a1a60 0x7fade009d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7fade546f640 10 monclient(hunting): picked mon.noname-a con 0x7fade00a1a60 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:05.733 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7fade546f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fade009f870 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7fade546f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fade00a0380 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7fade546f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade00a1260 con 0x7fade009f870 2024-09-15T04:05:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7fade546f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade009c870 con 0x7fade00a0380 2024-09-15T04:05:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7fade546f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7fade546f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade009cdf0 con 0x7fade00a1a60 2024-09-15T04:05:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7fade546f640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7fade546f640 10 monclient(hunting): authenticate will time out at 1129.374281s 2024-09-15T04:05:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fade00a1a60 0x7fade009d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7fadde7fc640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fade009f870 0x7fade009fc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58932/0 (socket says 172.21.15.110:58932) 2024-09-15T04:05:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7fadde7fc640 1 -- 172.21.15.110:0/4099679277 learned_addr learned my addr 172.21.15.110:0/4099679277 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 10 monclient(hunting): get_auth_request con 0x7fade00a1a60 auth_method 0 2024-09-15T04:05:05.734 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 10 monclient(hunting): handle_auth_done global_id 55185 payload 274 2024-09-15T04:05:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 1 -- 172.21.15.110:0/4099679277 >> v1:172.21.15.177:6789/0 conn(0x7fade00a0380 legacy=0x7fade0138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 1 -- 172.21.15.110:0/4099679277 >> v1:172.21.15.110:6789/0 conn(0x7fade009f870 legacy=0x7fade009fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 10 monclient: _finish_hunting 0 2024-09-15T04:05:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 1 monclient: found mon.noname-a 2024-09-15T04:05:05.735 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:05.736 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.730+0000 7faddeffd640 1 -- 172.21.15.110:0/4099679277 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade00a0c60 con 0x7fade00a1a60 2024-09-15T04:05:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7faddeffd640 1 -- 172.21.15.110:0/4099679277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fade00a1a60 msgr2=0x7fade009d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7faddeffd640 1 --2- 172.21.15.110:0/4099679277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fade00a1a60 0x7fade009d4f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fade00a2900 tx=0x7fadd0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294d7956957d45e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fadddffb640 1 -- 172.21.15.110:0/4099679277 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0007840 con 0x7fade00a1a60 2024-09-15T04:05:05.742 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fadddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fadddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fadddffb640 10 monclient: dump: 2024-09-15T04:05:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:05.743 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:05.743 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fadddffb640 10 monclient: _finish_auth 0 2024-09-15T04:05:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fadddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fadddffb640 1 -- 172.21.15.110:0/4099679277 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd002d050 con 0x7fade00a1a60 2024-09-15T04:05:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fade546f640 5 monclient: authenticate success, global_id 55185 2024-09-15T04:05:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fade546f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fadddffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fadddffb640 1 -- 172.21.15.110:0/4099679277 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0007bd0 con 0x7fade00a1a60 2024-09-15T04:05:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fade546f640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fade546f640 4 set_mon_vals no callback set 2024-09-15T04:05:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fadddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fadddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:05.744 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.733+0000 7fadddffb640 10 monclient: dump: 2024-09-15T04:05:05.745 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:05.745 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:05.745 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.745 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.745 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:05.745 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:05.745 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:05.745 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:05.745 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:05.745 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:05.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.734+0000 7fade546f640 10 monclient: shutdown 2024-09-15T04:05:05.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.734+0000 7fade546f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:05.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.734+0000 7fade546f640 1 -- 172.21.15.110:0/4099679277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fade00a1a60 msgr2=0x7fade009d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:05.745 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.734+0000 7fade546f640 1 --2- 172.21.15.110:0/4099679277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fade00a1a60 0x7fade009d4f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fade00a2900 tx=0x7fadd0009780 comp rx=0 tx=0).stop 2024-09-15T04:05:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.734+0000 7fade546f640 1 -- 172.21.15.110:0/4099679277 shutdown_connections 2024-09-15T04:05:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.734+0000 7fade546f640 1 --2- 172.21.15.110:0/4099679277 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fade00a1a60 0x7fade009d4f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.734+0000 7fade546f640 1 -- 172.21.15.110:0/4099679277 >> 172.21.15.110:0/4099679277 conn(0x7fade0063270 msgr2=0x7fade0063670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.734+0000 7fade546f640 1 -- 172.21.15.110:0/4099679277 shutdown_connections 2024-09-15T04:05:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.734+0000 7fade546f640 1 -- 172.21.15.110:0/4099679277 wait complete. 2024-09-15T04:05:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 10 monclient: build_initial_monmap 2024-09-15T04:05:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 10 monclient: monmap: 2024-09-15T04:05:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:05.746 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:05.747 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 1 Processor -- start 2024-09-15T04:05:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 1 -- start start 2024-09-15T04:05:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 10 monclient: init 2024-09-15T04:05:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T04:05:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fade009f870 0x7fade0134090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 10 monclient(hunting): picked mon.noname-c con 0x7fade009f870 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fade00a0380 0x7fade0136d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:05.747 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 10 monclient(hunting): picked mon.noname-b con 0x7fade00a0380 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fade00a1a60 0x7fade01372c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 10 monclient(hunting): picked mon.noname-a con 0x7fade00a1a60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade011c5b0 con 0x7fade00a1a60 2024-09-15T04:05:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade00a1450 con 0x7fade00a0380 2024-09-15T04:05:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade011bf60 con 0x7fade009f870 2024-09-15T04:05:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.735+0000 7fade546f640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7fade546f640 10 monclient(hunting): authenticate will time out at 1129.380005s 2024-09-15T04:05:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fade009f870 0x7fade0134090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddf7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fade00a1a60 0x7fade01372c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:05.748 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fade009f870 0x7fade0134090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35144/0 (socket says 172.21.15.110:35144) 2024-09-15T04:05:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 1 -- 172.21.15.110:0/2048352796 learned_addr learned my addr 172.21.15.110:0/2048352796 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 10 monclient(hunting): get_auth_request con 0x7fade009f870 auth_method 0 2024-09-15T04:05:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddf7fe640 10 monclient(hunting): get_auth_request con 0x7fade00a1a60 auth_method 0 2024-09-15T04:05:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:05.749 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7fadde7fc640 1 --2- 172.21.15.110:0/2048352796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fade00a0380 0x7fade0136d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7fadde7fc640 10 monclient(hunting): get_auth_request con 0x7fade00a0380 auth_method 0 2024-09-15T04:05:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7fadde7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7fadde7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7fadde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7fadde7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7fadde7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7fadde7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 10 monclient(hunting): handle_auth_done global_id 74135 payload 824 2024-09-15T04:05:05.750 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 1 -- 172.21.15.110:0/2048352796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fade00a0380 msgr2=0x7fade0136d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 1 --2- 172.21.15.110:0/2048352796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fade00a0380 0x7fade0136d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 1 -- 172.21.15.110:0/2048352796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fade00a1a60 msgr2=0x7fade01372c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 1 --2- 172.21.15.110:0/2048352796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fade00a1a60 0x7fade01372c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 10 monclient: _finish_hunting 0 2024-09-15T04:05:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 1 monclient: found mon.noname-c 2024-09-15T04:05:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddeffd640 1 -- 172.21.15.110:0/2048352796 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd0009a20 con 0x7fade009f870 2024-09-15T04:05:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.736+0000 7faddf7fe640 1 --2- 172.21.15.110:0/2048352796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fade00a1a60 0x7fade01372c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7faddeffd640 1 -- 172.21.15.110:0/2048352796 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fade009f870 msgr2=0x7fade0134090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7faddeffd640 1 --2- 172.21.15.110:0/2048352796 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fade009f870 0x7fade0134090 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fadd002ab30 tx=0x7fadd0037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fadc6ffd640 1 -- 172.21.15.110:0/2048352796 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0031a20 con 0x7fade009f870 2024-09-15T04:05:05.751 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fadc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fadc6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fadc6ffd640 10 monclient: dump: 2024-09-15T04:05:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:05.752 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fadc6ffd640 10 monclient: _finish_auth 0 2024-09-15T04:05:05.752 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fadc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fadc6ffd640 1 -- 172.21.15.110:0/2048352796 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd0040070 con 0x7fade009f870 2024-09-15T04:05:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fade546f640 5 monclient: authenticate success, global_id 74135 2024-09-15T04:05:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fadc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fadc6ffd640 1 -- 172.21.15.110:0/2048352796 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0031db0 con 0x7fade009f870 2024-09-15T04:05:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fadc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fadc6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fadc6ffd640 10 monclient: dump: 2024-09-15T04:05:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:05.753 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:05.754 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fade546f640 10 monclient: _renew_subs 2024-09-15T04:05:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fade546f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fade546f640 1 -- 172.21.15.110:0/2048352796 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade011c240 con 0x7fade009f870 2024-09-15T04:05:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7faddd7fa640 4 set_mon_vals no callback set 2024-09-15T04:05:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fade546f640 10 monclient: _renew_subs 2024-09-15T04:05:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fade546f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:05.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.738+0000 7fade546f640 1 -- 172.21.15.110:0/2048352796 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade01345d0 con 0x7fade009f870 2024-09-15T04:05:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.739+0000 7fadc6ffd640 1 -- 172.21.15.110:0/2048352796 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fadd0042030 con 0x7fade009f870 2024-09-15T04:05:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.739+0000 7fade546f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.739+0000 7fade546f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.739+0000 7fade546f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.739+0000 7fadc6ffd640 1 --2- 172.21.15.110:0/2048352796 >> v2:172.21.15.177:6812/28201 conn(0x7fadcc038700 0x7fadcc03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.739+0000 7fade546f640 1 -- 172.21.15.110:0/2048352796 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade0070950 con 0x7fade009f870 2024-09-15T04:05:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.739+0000 7fadc6ffd640 1 -- 172.21.15.110:0/2048352796 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fadd006cf00 con 0x7fade009f870 2024-09-15T04:05:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.739+0000 7fadde7fc640 1 --2- 172.21.15.110:0/2048352796 >> v2:172.21.15.177:6812/28201 conn(0x7fadcc038700 0x7fadcc03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:05.755 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.740+0000 7fadde7fc640 10 monclient: get_auth_request con 0x7fadcc038700 auth_method 0 2024-09-15T04:05:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.742+0000 7fadde7fc640 1 --2- 172.21.15.110:0/2048352796 >> v2:172.21.15.177:6812/28201 conn(0x7fadcc038700 0x7fadcc03abb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fade0138560 tx=0x7fadd4007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.744+0000 7fadc6ffd640 1 -- 172.21.15.110:0/2048352796 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fadd003b5e0 con 0x7fade009f870 2024-09-15T04:05:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.744+0000 7fadc6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:05.756 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.744+0000 7fadc6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:05.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.899+0000 7fade546f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T04:05:05.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.899+0000 7fade546f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T04:05:05.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.899+0000 7fade546f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:05.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.899+0000 7fade546f640 1 -- 172.21.15.110:0/2048352796 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fade0070950 con 0x7fade009f870 2024-09-15T04:05:05.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.900+0000 7fadc6ffd640 1 -- 172.21.15.110:0/2048352796 <== mon.2 v2:172.21.15.110:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v315) v1 ==== 74+0+10033 (secure 0 0 0) 0x7fadd0061090 con 0x7fade009f870 2024-09-15T04:05:05.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.900+0000 7fadc6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T04:05:05.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.900+0000 7fadc6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:05:05.902 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T04:05:05.902 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":315,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","created":"2024-09-15T03:58:28.265907+0000","modified":"2024-09-15T04:04:48.807362+0000","last_up_change":"2024-09-15T03:58:35.032799+0000","last_in_change":"2024-09-15T03:58:31.538597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":100,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:58:36.398831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:58:41.004777+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"105a3b3d-e410-47fb-ac54-6a5d5da4a46f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":28367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":28367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":28367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":28367}]},"public_addr":"172.21.15.110:6808/28367","cluster_addr":"172.21.15.110:6809/28367","heartbeat_back_addr":"172.21.15.110:6811/28367","heartbeat_front_addr":"172.21.15.110:6810/28367","state":["exists","up"]},{"osd":1,"uuid":"77f0f205-6c64-495a-8edd-043de180efa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":28369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":28369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":28369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":28369}]},"public_addr":"172.21.15.110:6800/28369","cluster_addr":"172.21.15.110:6801/28369","heartbeat_back_addr":"172.21.15.110:6803/28369","heartbeat_front_addr":"172.21.15.110:6802/28369","state":["exists","up"]},{"osd":2,"uuid":"e8a42031-303e-4ad0-9896-e5b84cc95cfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":28368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":28368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":28368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":28368}]},"public_addr":"172.21.15.110:6804/28368","cluster_addr":"172.21.15.110:6805/28368","heartbeat_back_addr":"172.21.15.110:6807/28368","heartbeat_front_addr":"172.21.15.110:6806/28368","state":["exists","up"]},{"osd":3,"uuid":"0d65db82-4a0e-4f31-a10a-dcd71dcb269f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":28628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6805","nonce":28628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6807","nonce":28628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":28628}]},"public_addr":"172.21.15.177:6804/28628","cluster_addr":"172.21.15.177:6805/28628","heartbeat_back_addr":"172.21.15.177:6807/28628","heartbeat_front_addr":"172.21.15.177:6806/28628","state":["exists","up"]},{"osd":4,"uuid":"e95e4f77-578d-4d03-bb4d-ee72b43bc227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":28629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6801","nonce":28629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6803","nonce":28629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":28629}]},"public_addr":"172.21.15.177:6800/28629","cluster_addr":"172.21.15.177:6801/28629","heartbeat_back_addr":"172.21.15.177:6803/28629","heartbeat_front_addr":"172.21.15.177:6802/28629","state":["exists","up"]},{"osd":5,"uuid":"f16ffee6-1e25-432c-b6c2-909a72ce24a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":28630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6809","nonce":28630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6811","nonce":28630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":28630}]},"public_addr":"172.21.15.177:6808/28630","cluster_addr":"172.21.15.177:6809/28630","heartbeat_back_addr":"172.21.15.177:6811/28630","heartbeat_front_addr":"172.21.15.177:6810/28630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.117560+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.137736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.130715+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.173663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.130009+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.172505+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T04:05:05.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.902+0000 7fade546f640 1 -- 172.21.15.110:0/2048352796 >> v2:172.21.15.177:6812/28201 conn(0x7fadcc038700 msgr2=0x7fadcc03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:05.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.902+0000 7fade546f640 1 --2- 172.21.15.110:0/2048352796 >> v2:172.21.15.177:6812/28201 conn(0x7fadcc038700 0x7fadcc03abb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fade0138560 tx=0x7fadd4007f00 comp rx=0 tx=0).stop 2024-09-15T04:05:05.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.902+0000 7fade546f640 10 monclient: shutdown 2024-09-15T04:05:05.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.902+0000 7fade546f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:05.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.902+0000 7fade546f640 1 -- 172.21.15.110:0/2048352796 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fade009f870 msgr2=0x7fade0134090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.902+0000 7fade546f640 1 --2- 172.21.15.110:0/2048352796 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fade009f870 0x7fade0134090 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fadd002ab30 tx=0x7fadd0037000 comp rx=0 tx=0).stop 2024-09-15T04:05:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.902+0000 7fade546f640 1 -- 172.21.15.110:0/2048352796 shutdown_connections 2024-09-15T04:05:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.902+0000 7fade546f640 1 --2- 172.21.15.110:0/2048352796 >> v2:172.21.15.177:6812/28201 conn(0x7fadcc038700 0x7fadcc03abb0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.902+0000 7fade546f640 1 --2- 172.21.15.110:0/2048352796 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fade00a1a60 0x7fade01372c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.902+0000 7fade546f640 1 --2- 172.21.15.110:0/2048352796 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fade00a0380 0x7fade0136d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.902+0000 7fade546f640 1 --2- 172.21.15.110:0/2048352796 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fade009f870 0x7fade0134090 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.902+0000 7fade546f640 1 -- 172.21.15.110:0/2048352796 >> 172.21.15.110:0/2048352796 conn(0x7fade0063270 msgr2=0x7fade0139ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.902+0000 7fade546f640 1 -- 172.21.15.110:0/2048352796 shutdown_connections 2024-09-15T04:05:05.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:05.902+0000 7fade546f640 1 -- 172.21.15.110:0/2048352796 wait complete. 2024-09-15T04:05:05.913 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T04:05:05.914 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T04:05:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.004+0000 7f85d1f9d640 10 monclient: get_monmap_and_config 2024-09-15T04:05:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.004+0000 7f85d1f9d640 10 monclient: build_initial_monmap 2024-09-15T04:05:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.004+0000 7f85d1f9d640 10 monclient: monmap: 2024-09-15T04:05:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:06.005324+0000 2024-09-15T04:05:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:06.005324+0000 2024-09-15T04:05:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:06.005 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:06.006 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:06.006 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.008 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 1 Processor -- start 2024-09-15T04:05:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 1 -- start start 2024-09-15T04:05:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 10 monclient: init 2024-09-15T04:05:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-15T04:05:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f85cc0a1bf0 0x7f85cc131810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f85cc0a1bf0 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f85cc09e780 0x7f85cc09eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f85cc09e780 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85cc09f760 0x7f85cc09fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f85cc09f760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85cc0a0c20 con 0x7f85cc09e780 2024-09-15T04:05:06.009 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85cc09cc90 con 0x7f85cc0a1bf0 2024-09-15T04:05:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc09c9b0 con 0x7f85cc09f760 2024-09-15T04:05:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85d1f9d640 10 monclient(hunting): authenticate will time out at 1129.649582s 2024-09-15T04:05:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cbfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85cc09f760 0x7f85cc09fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cbfff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85cc09f760 0x7f85cc09fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35146/0 (socket says 172.21.15.110:35146) 2024-09-15T04:05:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85caffd640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f85cc09e780 0x7f85cc09eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cbfff640 1 -- 172.21.15.110:0/3662583877 learned_addr learned my addr 172.21.15.110:0/3662583877 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cbfff640 10 monclient(hunting): get_auth_request con 0x7f85cc09f760 auth_method 0 2024-09-15T04:05:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.010 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85caffd640 10 monclient(hunting): get_auth_request con 0x7f85cc09e780 auth_method 0 2024-09-15T04:05:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85caffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cb7fe640 1 --2- 172.21.15.110:0/3662583877 >> v2:172.21.15.177:3300/0 conn(0x7f85cc0a1bf0 0x7f85cc131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cb7fe640 10 monclient(hunting): get_auth_request con 0x7f85cc0a1bf0 auth_method 0 2024-09-15T04:05:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.011 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.005+0000 7f85cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.006+0000 7f85caffd640 10 monclient(hunting): handle_auth_done global_id 55191 payload 274 2024-09-15T04:05:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.006+0000 7f85caffd640 1 -- 172.21.15.110:0/3662583877 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85cc09f760 msgr2=0x7f85cc09fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.006+0000 7f85caffd640 1 --2- 172.21.15.110:0/3662583877 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85cc09f760 0x7f85cc09fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.006+0000 7f85caffd640 1 -- 172.21.15.110:0/3662583877 >> v2:172.21.15.177:3300/0 conn(0x7f85cc0a1bf0 msgr2=0x7f85cc131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.006+0000 7f85caffd640 1 --2- 172.21.15.110:0/3662583877 >> v2:172.21.15.177:3300/0 conn(0x7f85cc0a1bf0 0x7f85cc131810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.006+0000 7f85caffd640 10 monclient: _finish_hunting 0 2024-09-15T04:05:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.006+0000 7f85caffd640 1 monclient: found mon.noname-a 2024-09-15T04:05:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.006+0000 7f85caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.006+0000 7f85caffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.006+0000 7f85caffd640 1 -- 172.21.15.110:0/3662583877 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85cc0a1fc0 con 0x7f85cc09e780 2024-09-15T04:05:06.012 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.006+0000 7f85cbfff640 1 --2- 172.21.15.110:0/3662583877 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85cc09f760 0x7f85cc09fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:06.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85caffd640 1 -- 172.21.15.110:0/3662583877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc09e780 msgr2=0x7f85cc09eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:06.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85caffd640 1 --2- 172.21.15.110:0/3662583877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc09e780 0x7f85cc09eb50 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f85cc072140 tx=0x7f85bc0090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e024f730b530fd92 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:06.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85ca7fc640 1 -- 172.21.15.110:0/3662583877 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85bc02ae90 con 0x7f85cc09e780 2024-09-15T04:05:06.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.018 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:06.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85ca7fc640 10 monclient: dump: 2024-09-15T04:05:06.019 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.019 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.019 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.019 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.019 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.019 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.019 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.019 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.019 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.019 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85ca7fc640 10 monclient: _finish_auth 0 2024-09-15T04:05:06.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:06.019 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85ca7fc640 1 -- 172.21.15.110:0/3662583877 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85bc032440 con 0x7f85cc09e780 2024-09-15T04:05:06.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85d1f9d640 5 monclient: authenticate success, global_id 55191 2024-09-15T04:05:06.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85d1f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:06.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:06.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85d1f9d640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:06.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85ca7fc640 1 -- 172.21.15.110:0/3662583877 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85bc0327e0 con 0x7f85cc09e780 2024-09-15T04:05:06.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85d1f9d640 4 set_mon_vals no callback set 2024-09-15T04:05:06.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:06.020 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.008+0000 7f85ca7fc640 10 monclient: dump: 2024-09-15T04:05:06.020 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.020 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.020 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.020 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.020 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.021 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.021 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.021 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.021 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.021 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.009+0000 7f85d1f9d640 10 monclient: shutdown 2024-09-15T04:05:06.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.009+0000 7f85d1f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:06.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.009+0000 7f85d1f9d640 1 -- 172.21.15.110:0/3662583877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc09e780 msgr2=0x7f85cc09eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.009+0000 7f85d1f9d640 1 --2- 172.21.15.110:0/3662583877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc09e780 0x7f85cc09eb50 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f85cc072140 tx=0x7f85bc0090f0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.009+0000 7f85d1f9d640 1 -- 172.21.15.110:0/3662583877 shutdown_connections 2024-09-15T04:05:06.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.009+0000 7f85d1f9d640 1 --2- 172.21.15.110:0/3662583877 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85cc09f760 0x7f85cc09fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.009+0000 7f85d1f9d640 1 --2- 172.21.15.110:0/3662583877 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc09e780 0x7f85cc09eb50 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.021 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.009+0000 7f85d1f9d640 1 --2- 172.21.15.110:0/3662583877 >> v2:172.21.15.177:3300/0 conn(0x7f85cc0a1bf0 0x7f85cc131810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.009+0000 7f85d1f9d640 1 -- 172.21.15.110:0/3662583877 >> 172.21.15.110:0/3662583877 conn(0x7f85cc062dd0 msgr2=0x7f85cc0631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:06.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.009+0000 7f85d1f9d640 1 -- 172.21.15.110:0/3662583877 shutdown_connections 2024-09-15T04:05:06.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.009+0000 7f85d1f9d640 1 -- 172.21.15.110:0/3662583877 wait complete. 2024-09-15T04:05:06.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.009+0000 7f85d1f9d640 10 monclient: build_initial_monmap 2024-09-15T04:05:06.022 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.009+0000 7f85d1f9d640 10 monclient: monmap: 2024-09-15T04:05:06.022 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:06.022 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:06.022 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:06.022 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:06.022 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:06.022 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.024 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:06.024 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:06.024 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:06.025 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 1 Processor -- start 2024-09-15T04:05:06.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 1 -- start start 2024-09-15T04:05:06.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 10 monclient: init 2024-09-15T04:05:06.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:06.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T04:05:06.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc09e780 0x7f85cc099ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f85cc09e780 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:06.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85cc09f760 0x7f85cc1307a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f85cc09f760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:06.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85cc0a1bf0 0x7f85cc140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f85cc0a1bf0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:06.025 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc11c790 con 0x7f85cc09e780 2024-09-15T04:05:06.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc09ae10 con 0x7f85cc0a1bf0 2024-09-15T04:05:06.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc09c250 con 0x7f85cc09f760 2024-09-15T04:05:06.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:06.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.010+0000 7f85d1f9d640 10 monclient(hunting): authenticate will time out at 1129.654943s 2024-09-15T04:05:06.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc09e780 0x7f85cc099ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85caffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85cc09f760 0x7f85cc1307a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc09e780 0x7f85cc099ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42798/0 (socket says 172.21.15.110:42798) 2024-09-15T04:05:06.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 1 -- 172.21.15.110:0/2036101468 learned_addr learned my addr 172.21.15.110:0/2036101468 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:06.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 10 monclient(hunting): get_auth_request con 0x7f85cc09e780 auth_method 0 2024-09-15T04:05:06.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.026 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85caffd640 10 monclient(hunting): get_auth_request con 0x7f85cc09f760 auth_method 0 2024-09-15T04:05:06.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85caffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cbfff640 1 --2- 172.21.15.110:0/2036101468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85cc0a1bf0 0x7f85cc140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cbfff640 10 monclient(hunting): get_auth_request con 0x7f85cc0a1bf0 auth_method 0 2024-09-15T04:05:06.027 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_done global_id 55194 payload 824 2024-09-15T04:05:06.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 1 -- 172.21.15.110:0/2036101468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85cc09f760 msgr2=0x7f85cc1307a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 1 --2- 172.21.15.110:0/2036101468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85cc09f760 0x7f85cc1307a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 1 -- 172.21.15.110:0/2036101468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85cc0a1bf0 msgr2=0x7f85cc140f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 1 --2- 172.21.15.110:0/2036101468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85cc0a1bf0 0x7f85cc140f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 10 monclient: _finish_hunting 0 2024-09-15T04:05:06.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 1 monclient: found mon.noname-a 2024-09-15T04:05:06.028 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:06.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:06.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85cb7fe640 1 -- 172.21.15.110:0/2036101468 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c4009ae0 con 0x7f85cc09e780 2024-09-15T04:05:06.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.011+0000 7f85caffd640 1 --2- 172.21.15.110:0/2036101468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85cc09f760 0x7f85cc1307a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:06.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85cb7fe640 1 -- 172.21.15.110:0/2036101468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc09e780 msgr2=0x7f85cc099ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:06.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85cb7fe640 1 --2- 172.21.15.110:0/2036101468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc09e780 0x7f85cc099ac0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f85b400d890 tx=0x7f85b4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:06.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85c37fe640 1 -- 172.21.15.110:0/2036101468 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b400de20 con 0x7f85cc09e780 2024-09-15T04:05:06.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85c37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:06.029 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85c37fe640 10 monclient: dump: 2024-09-15T04:05:06.029 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.029 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.029 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.029 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.029 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.030 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.030 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.030 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.030 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.030 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85c37fe640 10 monclient: _finish_auth 0 2024-09-15T04:05:06.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:06.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85c37fe640 1 -- 172.21.15.110:0/2036101468 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b400bae0 con 0x7f85cc09e780 2024-09-15T04:05:06.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85d1f9d640 5 monclient: authenticate success, global_id 55194 2024-09-15T04:05:06.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85d1f9d640 10 monclient: _renew_subs 2024-09-15T04:05:06.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85d1f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:06.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85d1f9d640 1 -- 172.21.15.110:0/2036101468 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85bc009390 con 0x7f85cc09e780 2024-09-15T04:05:06.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85c37fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:06.030 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85c37fe640 1 -- 172.21.15.110:0/2036101468 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b400be80 con 0x7f85cc09e780 2024-09-15T04:05:06.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:06.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85c37fe640 10 monclient: dump: 2024-09-15T04:05:06.031 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.031 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.031 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.031 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.031 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.031 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.031 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.031 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.031 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.031 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.031 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85d1f9d640 10 monclient: _renew_subs 2024-09-15T04:05:06.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85c9ffb640 4 set_mon_vals no callback set 2024-09-15T04:05:06.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85d1f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:06.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.013+0000 7f85d1f9d640 1 -- 172.21.15.110:0/2036101468 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85cc09a550 con 0x7f85cc09e780 2024-09-15T04:05:06.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.014+0000 7f85c37fe640 1 -- 172.21.15.110:0/2036101468 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f85b4014640 con 0x7f85cc09e780 2024-09-15T04:05:06.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.014+0000 7f85c37fe640 1 --2- 172.21.15.110:0/2036101468 >> v2:172.21.15.177:6812/28201 conn(0x7f85ac038730 0x7f85ac03abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.014+0000 7f85c37fe640 1 -- 172.21.15.110:0/2036101468 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f85b404bb90 con 0x7f85cc09e780 2024-09-15T04:05:06.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.014+0000 7f85d1f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:06.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.014+0000 7f85d1f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:06.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.014+0000 7f85d1f9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:06.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.014+0000 7f85d1f9d640 1 -- 172.21.15.110:0/2036101468 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85cc070950 con 0x7f85cc09e780 2024-09-15T04:05:06.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.014+0000 7f85caffd640 1 --2- 172.21.15.110:0/2036101468 >> v2:172.21.15.177:6812/28201 conn(0x7f85ac038730 0x7f85ac03abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.014+0000 7f85caffd640 10 monclient: get_auth_request con 0x7f85ac038730 auth_method 0 2024-09-15T04:05:06.032 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.017+0000 7f85caffd640 1 --2- 172.21.15.110:0/2036101468 >> v2:172.21.15.177:6812/28201 conn(0x7f85ac038730 0x7f85ac03abe0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0a2230 tx=0x7f85bc032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:06.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.019+0000 7f85c37fe640 1 -- 172.21.15.110:0/2036101468 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f85b401eb40 con 0x7f85cc09e780 2024-09-15T04:05:06.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.019+0000 7f85c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:06.033 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.019+0000 7f85c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:06.171 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.171+0000 7f85d1f9d640 1 -- 172.21.15.110:0/2036101468 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f85cc0a26c0 con 0x7f85ac038730 2024-09-15T04:05:06.174 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.173+0000 7f85c37fe640 1 -- 172.21.15.110:0/2036101468 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+52192 (secure 0 0 0) 0x7f85cc0a26c0 con 0x7f85ac038730 2024-09-15T04:05:06.174 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T04:05:06.175 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-09-15T04:05:06.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.175+0000 7f85d1f9d640 1 -- 172.21.15.110:0/2036101468 >> v2:172.21.15.177:6812/28201 conn(0x7f85ac038730 msgr2=0x7f85ac03abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.175+0000 7f85d1f9d640 1 --2- 172.21.15.110:0/2036101468 >> v2:172.21.15.177:6812/28201 conn(0x7f85ac038730 0x7f85ac03abe0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0a2230 tx=0x7f85bc032000 comp rx=0 tx=0).stop 2024-09-15T04:05:06.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.175+0000 7f85d1f9d640 10 monclient: shutdown 2024-09-15T04:05:06.176 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.175+0000 7f85d1f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:06.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.175+0000 7f85d1f9d640 1 -- 172.21.15.110:0/2036101468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc09e780 msgr2=0x7f85cc099ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.175+0000 7f85d1f9d640 1 --2- 172.21.15.110:0/2036101468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc09e780 0x7f85cc099ac0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f85b400d890 tx=0x7f85b4008040 comp rx=0 tx=0).stop 2024-09-15T04:05:06.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.175+0000 7f85d1f9d640 1 -- 172.21.15.110:0/2036101468 shutdown_connections 2024-09-15T04:05:06.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.175+0000 7f85d1f9d640 1 --2- 172.21.15.110:0/2036101468 >> v2:172.21.15.177:6812/28201 conn(0x7f85ac038730 0x7f85ac03abe0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.175+0000 7f85d1f9d640 1 --2- 172.21.15.110:0/2036101468 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85cc0a1bf0 0x7f85cc140f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.175+0000 7f85d1f9d640 1 --2- 172.21.15.110:0/2036101468 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f85cc09f760 0x7f85cc1307a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.175+0000 7f85d1f9d640 1 --2- 172.21.15.110:0/2036101468 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85cc09e780 0x7f85cc099ac0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.175+0000 7f85d1f9d640 1 -- 172.21.15.110:0/2036101468 >> 172.21.15.110:0/2036101468 conn(0x7f85cc062dd0 msgr2=0x7f85cc0a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:06.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.175+0000 7f85d1f9d640 1 -- 172.21.15.110:0/2036101468 shutdown_connections 2024-09-15T04:05:06.177 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.175+0000 7f85d1f9d640 1 -- 172.21.15.110:0/2036101468 wait complete. 2024-09-15T04:05:06.186 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-09-15T04:05:04.441714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562077696,"kb_used":4802164,"kb_used_data":557276556,"kb_used_omap":58124,"kb_used_meta":0,"kb_avail":557275532,"statfs":{"total":575567560704,"available":570650144768,"internally_reserved":1048576,"allocated":570651193344,"data_stored":570651193344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59519473,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6099,"num_objects":-50,"num_object_clones":0,"num_object_copies":-100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-50,"num_write_kb":-50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002590"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":281,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.812288+0000","last_change":"2024-09-15T03:58:44.072431+0000","last_active":"2024-09-15T04:04:47.812288+0000","last_peered":"2024-09-15T04:04:47.812288+0000","last_clean":"2024-09-15T04:04:47.812288+0000","last_became_active":"2024-09-15T03:58:42.157123+0000","last_became_peered":"2024-09-15T03:58:42.157123+0000","last_unstale":"2024-09-15T04:04:47.812288+0000","last_undegraded":"2024-09-15T04:04:47.812288+0000","last_fullsized":"2024-09-15T04:04:47.812288+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:46:37.797943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066518800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":302,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.808620+0000","last_change":"2024-09-15T03:58:44.072012+0000","last_active":"2024-09-15T04:04:47.808620+0000","last_peered":"2024-09-15T04:04:47.808620+0000","last_clean":"2024-09-15T04:04:47.808620+0000","last_became_active":"2024-09-15T03:58:42.069324+0000","last_became_peered":"2024-09-15T03:58:42.069324+0000","last_unstale":"2024-09-15T04:04:47.808620+0000","last_undegraded":"2024-09-15T04:04:47.808620+0000","last_fullsized":"2024-09-15T04:04:47.808620+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:44:04.033148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040535300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":281,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.811753+0000","last_change":"2024-09-15T03:58:44.072166+0000","last_active":"2024-09-15T04:04:47.811753+0000","last_peered":"2024-09-15T04:04:47.811753+0000","last_clean":"2024-09-15T04:04:47.811753+0000","last_became_active":"2024-09-15T03:58:42.155521+0000","last_became_peered":"2024-09-15T03:58:42.155521+0000","last_unstale":"2024-09-15T04:04:47.811753+0000","last_undegraded":"2024-09-15T04:04:47.811753+0000","last_fullsized":"2024-09-15T04:04:47.811753+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:42:01.920640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040527199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":302,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.808414+0000","last_change":"2024-09-15T03:58:44.072137+0000","last_active":"2024-09-15T04:04:47.808414+0000","last_peered":"2024-09-15T04:04:47.808414+0000","last_clean":"2024-09-15T04:04:47.808414+0000","last_became_active":"2024-09-15T03:58:42.155317+0000","last_became_peered":"2024-09-15T03:58:42.155317+0000","last_unstale":"2024-09-15T04:04:47.808414+0000","last_undegraded":"2024-09-15T04:04:47.808414+0000","last_fullsized":"2024-09-15T04:04:47.808414+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:55:28.163584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041321199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":275,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.813708+0000","last_change":"2024-09-15T03:58:44.074749+0000","last_active":"2024-09-15T04:04:47.813708+0000","last_peered":"2024-09-15T04:04:47.813708+0000","last_clean":"2024-09-15T04:04:47.813708+0000","last_became_active":"2024-09-15T03:58:42.071376+0000","last_became_peered":"2024-09-15T03:58:42.071376+0000","last_unstale":"2024-09-15T04:04:47.813708+0000","last_undegraded":"2024-09-15T04:04:47.813708+0000","last_fullsized":"2024-09-15T04:04:47.813708+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:42:52.640034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080084100000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":289,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.809641+0000","last_change":"2024-09-15T03:58:44.071453+0000","last_active":"2024-09-15T04:04:47.809641+0000","last_peered":"2024-09-15T04:04:47.809641+0000","last_clean":"2024-09-15T04:04:47.809641+0000","last_became_active":"2024-09-15T03:58:42.067012+0000","last_became_peered":"2024-09-15T03:58:42.067012+0000","last_unstale":"2024-09-15T04:04:47.809641+0000","last_undegraded":"2024-09-15T04:04:47.809641+0000","last_fullsized":"2024-09-15T04:04:47.809641+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:04:08.843151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040766700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":281,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.811226+0000","last_change":"2024-09-15T03:58:44.072155+0000","last_active":"2024-09-15T04:04:47.811226+0000","last_peered":"2024-09-15T04:04:47.811226+0000","last_clean":"2024-09-15T04:04:47.811226+0000","last_became_active":"2024-09-15T03:58:42.069300+0000","last_became_peered":"2024-09-15T03:58:42.069300+0000","last_unstale":"2024-09-15T04:04:47.811226+0000","last_undegraded":"2024-09-15T04:04:47.811226+0000","last_fullsized":"2024-09-15T04:04:47.811226+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:28:34.591535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036612400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":274,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.814025+0000","last_change":"2024-09-15T03:58:44.074233+0000","last_active":"2024-09-15T04:04:47.814025+0000","last_peered":"2024-09-15T04:04:47.814025+0000","last_clean":"2024-09-15T04:04:47.814025+0000","last_became_active":"2024-09-15T03:58:42.071652+0000","last_became_peered":"2024-09-15T03:58:42.071652+0000","last_unstale":"2024-09-15T04:04:47.814025+0000","last_undegraded":"2024-09-15T04:04:47.814025+0000","last_fullsized":"2024-09-15T04:04:47.814025+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:31:44.288660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030212199999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":369,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.811314+0000","last_change":"2024-09-15T03:58:38.057397+0000","last_active":"2024-09-15T04:04:47.811314+0000","last_peered":"2024-09-15T04:04:47.811314+0000","last_clean":"2024-09-15T04:04:47.811314+0000","last_became_active":"2024-09-15T03:58:38.056673+0000","last_became_peered":"2024-09-15T03:58:38.056673+0000","last_unstale":"2024-09-15T04:04:47.811314+0000","last_undegraded":"2024-09-15T04:04:47.811314+0000","last_fullsized":"2024-09-15T04:04:47.811314+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_clean_scrub_stamp":"2024-09-15T03:58:37.036429+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:38:58.792683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":800116,"kb_used_data":92879620,"kb_used_omap":9352,"kb_used_meta":0,"kb_avail":92879500,"statfs":{"total":95927926784,"available":95108608000,"internally_reserved":122880,"allocated":95108730880,"data_stored":95108730880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9577085,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48399999999999999,"15min":0.64000000000000001},"min":{"1min":0.32800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63800000000000001,"5min":0.70399999999999996,"15min":5.0810000000000004},"last":0.499},{"interface":"front","average":{"1min":0.622,"5min":0.50900000000000001,"15min":0.46100000000000002},"min":{"1min":0.33700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":0.47499999999999998}]},{"osd":1,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.54600000000000004,"15min":0.47899999999999998},"min":{"1min":0.26500000000000001,"5min":0.248,"15min":0.248},"max":{"1min":2.6720000000000002,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51700000000000002,"15min":0.46100000000000002},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66200000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.46700000000000003}]},{"osd":2,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.61799999999999999,"5min":0.63900000000000001,"15min":0.71099999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.51500000000000001,"15min":0.73799999999999999},"min":{"1min":0.38600000000000001,"5min":0.314,"15min":0.314},"max":{"1min":2.6320000000000001,"5min":2.6320000000000001,"15min":7.3520000000000003},"last":0.436}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.24199999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.497,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.28100000000000003,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.57699999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.48899999999999999}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.38900000000000001,"15min":0.56999999999999995},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.5059999999999998,"5min":2.5059999999999998,"15min":5.1879999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.38200000000000001,"15min":0.57499999999999996},"min":{"1min":0.28399999999999997,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.51800000000000002,"5min":0.60699999999999998,"15min":5.1749999999999998},"last":0.41199999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640335,"num_pgs":18,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":800272,"kb_used_data":92879624,"kb_used_omap":9866,"kb_used_meta":0,"kb_avail":92879344,"statfs":{"total":95927926784,"available":95108448256,"internally_reserved":286720,"allocated":95108734976,"data_stored":95108734976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10103336,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.78200000000000003},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.70299999999999996,"5min":1.3959999999999999,"15min":9.9429999999999996},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.51000000000000001,"15min":0.81699999999999995},"min":{"1min":0.26200000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.68500000000000005,"5min":1.272,"15min":11.509},"last":0.47499999999999998}]},{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56299999999999994,"15min":0.53700000000000003},"min":{"1min":0.311,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.76200000000000001,"5min":2.867,"15min":2.867},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56999999999999995,"15min":0.57699999999999996},"min":{"1min":0.33400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":1.7689999999999999,"5min":3.923,"15min":3.923},"last":0.45700000000000002}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51800000000000002,"15min":0.496},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69999999999999996,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.71799999999999997},"min":{"1min":0.35899999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.73999999999999999,"5min":1.3540000000000001,"15min":7.5140000000000002},"last":0.51200000000000001}]},{"osd":3,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.40000000000000002,"15min":0.39200000000000002},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.53300000000000003,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.435},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41999999999999998,"15min":0.42699999999999999},"min":{"1min":0.26600000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.55200000000000005,"5min":2.339,"15min":2.339},"last":0.42099999999999999}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44700000000000001,"15min":0.73599999999999999},"min":{"1min":0.33500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.5,"5min":3.7080000000000002,"15min":11.814},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.438,"15min":0.42599999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.53300000000000003,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.40600000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640335,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":800712,"kb_used_data":92879036,"kb_used_omap":9496,"kb_used_meta":0,"kb_avail":92878904,"statfs":{"total":95927926784,"available":95107997696,"internally_reserved":135168,"allocated":95108132864,"data_stored":95108132864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9724326,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.51600000000000001,"15min":0.48699999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.19900000000000001},"max":{"1min":0.63600000000000001,"5min":3.1429999999999998,"15min":3.1429999999999998},"last":0.498},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48699999999999999,"15min":7.2050000000000001},"min":{"1min":0.33700000000000002,"5min":0.27900000000000003,"15min":0.23499999999999999},"max":{"1min":0.73699999999999999,"5min":0.76200000000000001,"15min":192.55600000000001},"last":0.48099999999999998}]},{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52100000000000002,"15min":7.2519999999999998},"min":{"1min":0.32300000000000001,"5min":0.30399999999999999,"15min":0.22600000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":192.50899999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55200000000000005,"15min":0.53200000000000003},"min":{"1min":0.32500000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.76700000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.54900000000000004}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.501,"15min":0.64500000000000002},"min":{"1min":0.35399999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.624,"5min":0.79500000000000004,"15min":5.0419999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.73199999999999998},"min":{"1min":0.40100000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":0.77700000000000002,"15min":7.2930000000000001},"last":0.36599999999999999}]},{"osd":4,"last update":"Sun Sep 15 04:04:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.36299999999999999,"15min":0.35499999999999998},"min":{"1min":0.223,"5min":0.222,"15min":0.222},"max":{"1min":0.46000000000000002,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.41299999999999998,"15min":0.38700000000000001},"min":{"1min":0.34499999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67300000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.379}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39200000000000002},"min":{"1min":0.27100000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.625,"5min":0.67800000000000005,"15min":0.69999999999999996},"last":0.314},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.39400000000000002,"15min":0.40400000000000003},"min":{"1min":0.30399999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.5,"5min":0.625,"15min":0.625},"last":0.35499999999999998}]}]},{"osd":2,"up_from":11,"seq":47244640335,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":800120,"kb_used_data":92879616,"kb_used_omap":9702,"kb_used_meta":0,"kb_avail":92879496,"statfs":{"total":95927926784,"available":95108603904,"internally_reserved":122880,"allocated":95108726784,"data_stored":95108726784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9935219,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.316,"15min":0.34000000000000002},"min":{"1min":0.20699999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.47699999999999998,"5min":0.502,"15min":0.57699999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.33000000000000002,"15min":0.36399999999999999},"min":{"1min":0.223,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.44500000000000001,"5min":0.56999999999999995,"15min":0.65100000000000002},"last":0.38700000000000001}]},{"osd":1,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.34599999999999997},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48999999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.433},{"interface":"front","average":{"1min":0.38,"5min":0.35899999999999999,"15min":0.372},"min":{"1min":0.28499999999999998,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54400000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.45700000000000002}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.67100000000000004,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72299999999999998,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.59399999999999997}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.66200000000000003,"5min":0.745,"15min":0.745},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54100000000000004,"15min":0.63500000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.754,"5min":1.208,"15min":3.1339999999999999},"last":0.61199999999999999}]},{"osd":5,"last update":"Sun Sep 15 04:04:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.35999999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.65400000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73199999999999998,"5min":0.77800000000000002,"15min":2.9500000000000002},"last":0.63700000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640335,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":800192,"kb_used_data":92879616,"kb_used_omap":9674,"kb_used_meta":0,"kb_avail":92879424,"statfs":{"total":95927926784,"available":95108530176,"internally_reserved":196608,"allocated":95108726784,"data_stored":95108726784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9906910,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34499999999999997,"15min":0.34300000000000003},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.52700000000000002,"5min":1.101,"15min":1.101},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.35299999999999998,"15min":0.35099999999999998},"min":{"1min":0.26400000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.497,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.31900000000000001}]},{"osd":2,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.376,"15min":0.36699999999999999},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":0.55600000000000005,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.374,"5min":0.36199999999999999,"15min":0.374},"min":{"1min":0.26300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.5,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.33900000000000002}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52200000000000002,"15min":0.54800000000000004},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.247},"max":{"1min":0.65400000000000003,"5min":1.087,"15min":1.087},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.54900000000000004},"min":{"1min":0.35999999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68500000000000005,"5min":1.032,"15min":1.032},"last":0.44}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.57699999999999996,"15min":0.56100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66400000000000003,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.623,"15min":0.59399999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68300000000000005,"5min":4.2930000000000001,"15min":4.2930000000000001},"last":0.50900000000000001}]},{"osd":5,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.33000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.72899999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.36899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70899999999999996,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.53400000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640335,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":800752,"kb_used_data":92879044,"kb_used_omap":10031,"kb_used_meta":0,"kb_avail":92878864,"statfs":{"total":95927926784,"available":95107956736,"internally_reserved":184320,"allocated":95108141056,"data_stored":95108141056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10272597,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"min":{"1min":0.246,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.49099999999999999,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.35999999999999999},"min":{"1min":0.309,"5min":0.193,"15min":0.193},"max":{"1min":0.496,"5min":0.504,"15min":0.71199999999999997},"last":0.23999999999999999}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.34799999999999998,"15min":0.376},"min":{"1min":0.32400000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52000000000000002,"5min":0.67500000000000004,"15min":0.78300000000000003},"last":0.255},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.38800000000000001},"min":{"1min":0.29199999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.54100000000000004,"5min":0.65200000000000002,"15min":0.76000000000000001},"last":0.27500000000000002}]},{"osd":3,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.43099999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.80100000000000005,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.379},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72299999999999998,"5min":0.85599999999999998,"15min":0.88200000000000001},"last":0.42699999999999999}]},{"osd":4,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.46300000000000002,"5min":0.30099999999999999,"15min":0.29599999999999999},"max":{"1min":0.84599999999999997,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.57199999999999995,"15min":0.59599999999999997},"min":{"1min":0.53100000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.40699999999999997}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52800000000000002,"15min":0.54800000000000004},"min":{"1min":0.44500000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.56200000000000006,"15min":0.54800000000000004},"min":{"1min":0.47399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76000000000000001,"5min":1.044,"15min":1.044},"last":0.44400000000000001}]}]}],"pool_statfs":[]}} 2024-09-15T04:05:06.188 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T04:05:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.281+0000 7f9ecee17640 10 monclient: get_monmap_and_config 2024-09-15T04:05:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.281+0000 7f9ecee17640 10 monclient: build_initial_monmap 2024-09-15T04:05:06.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.281+0000 7f9ecee17640 10 monclient: monmap: 2024-09-15T04:05:06.283 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:06.283 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.283 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:06.282804+0000 2024-09-15T04:05:06.283 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:06.282804+0000 2024-09-15T04:05:06.283 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:06.283 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.283 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:06.283 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:06.283 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:06.283 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:06.283 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:06.283 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.282+0000 7f9ecee17640 1 Processor -- start 2024-09-15T04:05:06.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.282+0000 7f9ecee17640 1 -- start start 2024-09-15T04:05:06.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.282+0000 7f9ecee17640 10 monclient: init 2024-09-15T04:05:06.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ecee17640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:06.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ecee17640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-15T04:05:06.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ecee17640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809bca0 0x7f9ec8096ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ecee17640 10 monclient(hunting): picked mon.noname-b con 0x7f9ec809bca0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:06.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ecee17640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9ec809ccf0 0x7f9ec809c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ecee17640 10 monclient(hunting): picked mon.noname-a con 0x7f9ec809ccf0 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:06.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ecee17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ec809c5b0 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:06.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ecee17640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec80966b0 con 0x7f9ec809c5b0 2024-09-15T04:05:06.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ecee17640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ecee17640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec8096460 con 0x7f9ec809ccf0 2024-09-15T04:05:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ecee17640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ecee17640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec8095e10 con 0x7f9ec809bca0 2024-09-15T04:05:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ecee17640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ecee17640 10 monclient(hunting): authenticate will time out at 1129.927088s 2024-09-15T04:05:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809bca0 0x7f9ec8096ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ec7fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9ec809ccf0 0x7f9ec809c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ec7fff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9ec809ccf0 0x7f9ec809c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42802/0 (socket says 172.21.15.110:42802) 2024-09-15T04:05:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809bca0 0x7f9ec8096ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35164/0 (socket says 172.21.15.110:35164) 2024-09-15T04:05:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ec7fff640 1 -- 172.21.15.110:0/3900000071 learned_addr learned my addr 172.21.15.110:0/3900000071 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ec7fff640 10 monclient(hunting): get_auth_request con 0x7f9ec809ccf0 auth_method 0 2024-09-15T04:05:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ec7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 10 monclient(hunting): get_auth_request con 0x7f9ec809bca0 auth_method 0 2024-09-15T04:05:06.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ec7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ec7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ec7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 10 monclient(hunting): handle_auth_done global_id 74144 payload 274 2024-09-15T04:05:06.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 1 -- 172.21.15.110:0/3900000071 >> v2:172.21.15.110:3300/0 conn(0x7f9ec809ccf0 msgr2=0x7f9ec809c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:06.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 1 --2- 172.21.15.110:0/3900000071 >> v2:172.21.15.110:3300/0 conn(0x7f9ec809ccf0 0x7f9ec809c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 1 -- 172.21.15.110:0/3900000071 >> v1:172.21.15.177:6789/0 conn(0x7f9ec809c5b0 legacy=0x7f9ec8137ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:06.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 10 monclient: _finish_hunting 0 2024-09-15T04:05:06.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 1 monclient: found mon.noname-b 2024-09-15T04:05:06.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:06.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:05:06.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9eccbb3640 1 -- 172.21.15.110:0/3900000071 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec809d5d0 con 0x7f9ec809bca0 2024-09-15T04:05:06.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.283+0000 7f9ec7fff640 1 --2- 172.21.15.110:0/3900000071 >> v2:172.21.15.110:3300/0 conn(0x7f9ec809ccf0 0x7f9ec809c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:06.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9eccbb3640 1 -- 172.21.15.110:0/3900000071 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809bca0 msgr2=0x7f9ec8096ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:06.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9eccbb3640 1 --2- 172.21.15.110:0/3900000071 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809bca0 0x7f9ec8096ad0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80a2270 tx=0x7f9ebc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aca88727a5e30636 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:06.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ec77fe640 1 -- 172.21.15.110:0/3900000071 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc02adb0 con 0x7f9ec809bca0 2024-09-15T04:05:06.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ec77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ec77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:05:06.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ec77fe640 10 monclient: dump: 2024-09-15T04:05:06.295 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.295 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.296 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.296 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.296 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.296 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.296 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.296 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.296 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.296 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ec77fe640 10 monclient: _finish_auth 0 2024-09-15T04:05:06.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ec77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:06.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ec77fe640 1 -- 172.21.15.110:0/3900000071 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ebc032440 con 0x7f9ec809bca0 2024-09-15T04:05:06.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ecee17640 5 monclient: authenticate success, global_id 74144 2024-09-15T04:05:06.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ecee17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ec77fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ecee17640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ecee17640 4 set_mon_vals no callback set 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ec77fe640 1 -- 172.21.15.110:0/3900000071 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc0327e0 con 0x7f9ec809bca0 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ec77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ec77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ec77fe640 10 monclient: dump: 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.297 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.298 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ecee17640 10 monclient: shutdown 2024-09-15T04:05:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ecee17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ecee17640 1 -- 172.21.15.110:0/3900000071 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809bca0 msgr2=0x7f9ec8096ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.286+0000 7f9ecee17640 1 --2- 172.21.15.110:0/3900000071 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809bca0 0x7f9ec8096ad0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80a2270 tx=0x7f9ebc0090f0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.287+0000 7f9ecee17640 1 -- 172.21.15.110:0/3900000071 shutdown_connections 2024-09-15T04:05:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.287+0000 7f9ecee17640 1 --2- 172.21.15.110:0/3900000071 >> v2:172.21.15.110:3300/0 conn(0x7f9ec809ccf0 0x7f9ec809c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.287+0000 7f9ecee17640 1 --2- 172.21.15.110:0/3900000071 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809bca0 0x7f9ec8096ad0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.287+0000 7f9ecee17640 1 -- 172.21.15.110:0/3900000071 >> 172.21.15.110:0/3900000071 conn(0x7f9ec80678d0 msgr2=0x7f9ec80a12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.287+0000 7f9ecee17640 1 -- 172.21.15.110:0/3900000071 shutdown_connections 2024-09-15T04:05:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.287+0000 7f9ecee17640 1 -- 172.21.15.110:0/3900000071 wait complete. 2024-09-15T04:05:06.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.287+0000 7f9ecee17640 10 monclient: build_initial_monmap 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.287+0000 7f9ecee17640 10 monclient: monmap: 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.287+0000 7f9ecee17640 1 Processor -- start 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 1 -- start start 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 10 monclient: init 2024-09-15T04:05:06.299 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T04:05:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ec809bca0 0x7f9ec811e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 10 monclient(hunting): picked mon.noname-b con 0x7f9ec809bca0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809c5b0 0x7f9ec811ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 10 monclient(hunting): picked mon.noname-c con 0x7f9ec809c5b0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ec809ccf0 0x7f9ec811f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 10 monclient(hunting): picked mon.noname-a con 0x7f9ec809ccf0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec8120e50 con 0x7f9ec809ccf0 2024-09-15T04:05:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec81220b0 con 0x7f9ec809bca0 2024-09-15T04:05:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec8122300 con 0x7f9ec809c5b0 2024-09-15T04:05:06.300 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecee17640 10 monclient(hunting): authenticate will time out at 1129.932489s 2024-09-15T04:05:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809c5b0 0x7f9ec811ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecd3b4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ec809ccf0 0x7f9ec811f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809c5b0 0x7f9ec811ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35174/0 (socket says 172.21.15.110:35174) 2024-09-15T04:05:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 1 -- 172.21.15.110:0/2022481846 learned_addr learned my addr 172.21.15.110:0/2022481846 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 10 monclient(hunting): get_auth_request con 0x7f9ec809c5b0 auth_method 0 2024-09-15T04:05:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecd3b4640 10 monclient(hunting): get_auth_request con 0x7f9ec809ccf0 auth_method 0 2024-09-15T04:05:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecd3b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecd3b4640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecd3b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.301 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecd3b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecd3b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecd3b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9eccbb3640 1 --2- 172.21.15.110:0/2022481846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ec809bca0 0x7f9ec811e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9eccbb3640 10 monclient(hunting): get_auth_request con 0x7f9ec809bca0 auth_method 0 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9eccbb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9eccbb3640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9eccbb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 10 monclient(hunting): handle_auth_done global_id 74147 payload 824 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 1 -- 172.21.15.110:0/2022481846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ec809bca0 msgr2=0x7f9ec811e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 1 --2- 172.21.15.110:0/2022481846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ec809bca0 0x7f9ec811e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 1 -- 172.21.15.110:0/2022481846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ec809ccf0 msgr2=0x7f9ec811f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 1 --2- 172.21.15.110:0/2022481846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ec809ccf0 0x7f9ec811f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 10 monclient: _finish_hunting 0 2024-09-15T04:05:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 1 monclient: found mon.noname-c 2024-09-15T04:05:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ec7fff640 1 -- 172.21.15.110:0/2022481846 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb4009450 con 0x7f9ec809c5b0 2024-09-15T04:05:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9eccbb3640 1 --2- 172.21.15.110:0/2022481846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ec809bca0 0x7f9ec811e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T04:05:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.288+0000 7f9ecd3b4640 1 --2- 172.21.15.110:0/2022481846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ec809ccf0 0x7f9ec811f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec7fff640 1 -- 172.21.15.110:0/2022481846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809c5b0 msgr2=0x7f9ec811ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec7fff640 1 --2- 172.21.15.110:0/2022481846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809c5b0 0x7f9ec811ed30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400ecb0 tx=0x7f9eb40091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec4ff9640 1 -- 172.21.15.110:0/2022481846 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb400cdd0 con 0x7f9ec809c5b0 2024-09-15T04:05:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:06.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec4ff9640 10 monclient: dump: 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec4ff9640 10 monclient: _finish_auth 0 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec4ff9640 1 -- 172.21.15.110:0/2022481846 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb4013470 con 0x7f9ec809c5b0 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ecee17640 5 monclient: authenticate success, global_id 74147 2024-09-15T04:05:06.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ecee17640 10 monclient: _renew_subs 2024-09-15T04:05:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ecee17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ecee17640 1 -- 172.21.15.110:0/2022481846 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ebc009390 con 0x7f9ec809c5b0 2024-09-15T04:05:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec4ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec4ff9640 1 -- 172.21.15.110:0/2022481846 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4013810 con 0x7f9ec809c5b0 2024-09-15T04:05:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec4ff9640 10 monclient: dump: 2024-09-15T04:05:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.305 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ec6ffd640 4 set_mon_vals no callback set 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ecee17640 10 monclient: _renew_subs 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ecee17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.291+0000 7f9ecee17640 1 -- 172.21.15.110:0/2022481846 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec809b460 con 0x7f9ec809c5b0 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.292+0000 7f9ec4ff9640 1 -- 172.21.15.110:0/2022481846 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f9eb4013bc0 con 0x7f9ec809c5b0 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.292+0000 7f9ec4ff9640 1 --2- 172.21.15.110:0/2022481846 >> v2:172.21.15.177:6812/28201 conn(0x7f9eac038700 0x7f9eac03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.292+0000 7f9ecee17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.292+0000 7f9ecee17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.292+0000 7f9ecee17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.292+0000 7f9ecee17640 1 -- 172.21.15.110:0/2022481846 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec8122440 con 0x7f9ec809c5b0 2024-09-15T04:05:06.306 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.292+0000 7f9ec4ff9640 1 -- 172.21.15.110:0/2022481846 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9eb404ba70 con 0x7f9ec809c5b0 2024-09-15T04:05:06.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.292+0000 7f9eccbb3640 1 --2- 172.21.15.110:0/2022481846 >> v2:172.21.15.177:6812/28201 conn(0x7f9eac038700 0x7f9eac03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.292+0000 7f9eccbb3640 10 monclient: get_auth_request con 0x7f9eac038700 auth_method 0 2024-09-15T04:05:06.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.295+0000 7f9eccbb3640 1 --2- 172.21.15.110:0/2022481846 >> v2:172.21.15.177:6812/28201 conn(0x7f9eac038700 0x7f9eac03abb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80637b0 tx=0x7f9ebc032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:06.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.297+0000 7f9ec4ff9640 1 -- 172.21.15.110:0/2022481846 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f9ec8122440 con 0x7f9ec809c5b0 2024-09-15T04:05:06.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.297+0000 7f9ec4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:06.307 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.297+0000 7f9ec4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:06.448 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.447+0000 7f9ecee17640 1 -- 172.21.15.110:0/2022481846 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9ec8122440 con 0x7f9eac038700 2024-09-15T04:05:06.450 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.449+0000 7f9ec4ff9640 1 -- 172.21.15.110:0/2022481846 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+52166 (secure 0 0 0) 0x7f9ec8122440 con 0x7f9eac038700 2024-09-15T04:05:06.451 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T04:05:06.451 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-09-15T04:05:06.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.451+0000 7f9ecee17640 1 -- 172.21.15.110:0/2022481846 >> v2:172.21.15.177:6812/28201 conn(0x7f9eac038700 msgr2=0x7f9eac03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.451+0000 7f9ecee17640 1 --2- 172.21.15.110:0/2022481846 >> v2:172.21.15.177:6812/28201 conn(0x7f9eac038700 0x7f9eac03abb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80637b0 tx=0x7f9ebc032000 comp rx=0 tx=0).stop 2024-09-15T04:05:06.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.452+0000 7f9ecee17640 10 monclient: shutdown 2024-09-15T04:05:06.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.452+0000 7f9ecee17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:06.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.452+0000 7f9ecee17640 1 -- 172.21.15.110:0/2022481846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809c5b0 msgr2=0x7f9ec811ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.452+0000 7f9ecee17640 1 --2- 172.21.15.110:0/2022481846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809c5b0 0x7f9ec811ed30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9eb400ecb0 tx=0x7f9eb40091d0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.452+0000 7f9ecee17640 1 -- 172.21.15.110:0/2022481846 shutdown_connections 2024-09-15T04:05:06.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.452+0000 7f9ecee17640 1 --2- 172.21.15.110:0/2022481846 >> v2:172.21.15.177:6812/28201 conn(0x7f9eac038700 0x7f9eac03abb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.452+0000 7f9ecee17640 1 --2- 172.21.15.110:0/2022481846 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ec809ccf0 0x7f9ec811f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.452+0000 7f9ecee17640 1 --2- 172.21.15.110:0/2022481846 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f9ec809c5b0 0x7f9ec811ed30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.452+0000 7f9ecee17640 1 --2- 172.21.15.110:0/2022481846 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9ec809bca0 0x7f9ec811e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.453 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.452+0000 7f9ecee17640 1 -- 172.21.15.110:0/2022481846 >> 172.21.15.110:0/2022481846 conn(0x7f9ec80678d0 msgr2=0x7f9ec8098b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:06.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.452+0000 7f9ecee17640 1 -- 172.21.15.110:0/2022481846 shutdown_connections 2024-09-15T04:05:06.454 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.452+0000 7f9ecee17640 1 -- 172.21.15.110:0/2022481846 wait complete. 2024-09-15T04:05:06.463 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-09-15T04:05:06.442175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562077696,"kb_used":4800656,"kb_used_data":557277040,"kb_used_omap":58127,"kb_used_meta":0,"kb_avail":557277040,"statfs":{"total":575567560704,"available":570651688960,"internally_reserved":0,"allocated":570651688960,"data_stored":570651688960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59522189,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6099,"num_objects":-50,"num_object_clones":0,"num_object_copies":-100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-50,"num_write_kb":-50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002544"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":281,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.812288+0000","last_change":"2024-09-15T03:58:44.072431+0000","last_active":"2024-09-15T04:04:47.812288+0000","last_peered":"2024-09-15T04:04:47.812288+0000","last_clean":"2024-09-15T04:04:47.812288+0000","last_became_active":"2024-09-15T03:58:42.157123+0000","last_became_peered":"2024-09-15T03:58:42.157123+0000","last_unstale":"2024-09-15T04:04:47.812288+0000","last_undegraded":"2024-09-15T04:04:47.812288+0000","last_fullsized":"2024-09-15T04:04:47.812288+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:46:37.797943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066518800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":302,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.808620+0000","last_change":"2024-09-15T03:58:44.072012+0000","last_active":"2024-09-15T04:04:47.808620+0000","last_peered":"2024-09-15T04:04:47.808620+0000","last_clean":"2024-09-15T04:04:47.808620+0000","last_became_active":"2024-09-15T03:58:42.069324+0000","last_became_peered":"2024-09-15T03:58:42.069324+0000","last_unstale":"2024-09-15T04:04:47.808620+0000","last_undegraded":"2024-09-15T04:04:47.808620+0000","last_fullsized":"2024-09-15T04:04:47.808620+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:44:04.033148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040535300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":281,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.811753+0000","last_change":"2024-09-15T03:58:44.072166+0000","last_active":"2024-09-15T04:04:47.811753+0000","last_peered":"2024-09-15T04:04:47.811753+0000","last_clean":"2024-09-15T04:04:47.811753+0000","last_became_active":"2024-09-15T03:58:42.155521+0000","last_became_peered":"2024-09-15T03:58:42.155521+0000","last_unstale":"2024-09-15T04:04:47.811753+0000","last_undegraded":"2024-09-15T04:04:47.811753+0000","last_fullsized":"2024-09-15T04:04:47.811753+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:42:01.920640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040527199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":302,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.808414+0000","last_change":"2024-09-15T03:58:44.072137+0000","last_active":"2024-09-15T04:04:47.808414+0000","last_peered":"2024-09-15T04:04:47.808414+0000","last_clean":"2024-09-15T04:04:47.808414+0000","last_became_active":"2024-09-15T03:58:42.155317+0000","last_became_peered":"2024-09-15T03:58:42.155317+0000","last_unstale":"2024-09-15T04:04:47.808414+0000","last_undegraded":"2024-09-15T04:04:47.808414+0000","last_fullsized":"2024-09-15T04:04:47.808414+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:55:28.163584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041321199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":275,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.813708+0000","last_change":"2024-09-15T03:58:44.074749+0000","last_active":"2024-09-15T04:04:47.813708+0000","last_peered":"2024-09-15T04:04:47.813708+0000","last_clean":"2024-09-15T04:04:47.813708+0000","last_became_active":"2024-09-15T03:58:42.071376+0000","last_became_peered":"2024-09-15T03:58:42.071376+0000","last_unstale":"2024-09-15T04:04:47.813708+0000","last_undegraded":"2024-09-15T04:04:47.813708+0000","last_fullsized":"2024-09-15T04:04:47.813708+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:42:52.640034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080084100000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":289,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.809641+0000","last_change":"2024-09-15T03:58:44.071453+0000","last_active":"2024-09-15T04:04:47.809641+0000","last_peered":"2024-09-15T04:04:47.809641+0000","last_clean":"2024-09-15T04:04:47.809641+0000","last_became_active":"2024-09-15T03:58:42.067012+0000","last_became_peered":"2024-09-15T03:58:42.067012+0000","last_unstale":"2024-09-15T04:04:47.809641+0000","last_undegraded":"2024-09-15T04:04:47.809641+0000","last_fullsized":"2024-09-15T04:04:47.809641+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:04:08.843151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040766700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":281,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.811226+0000","last_change":"2024-09-15T03:58:44.072155+0000","last_active":"2024-09-15T04:04:47.811226+0000","last_peered":"2024-09-15T04:04:47.811226+0000","last_clean":"2024-09-15T04:04:47.811226+0000","last_became_active":"2024-09-15T03:58:42.069300+0000","last_became_peered":"2024-09-15T03:58:42.069300+0000","last_unstale":"2024-09-15T04:04:47.811226+0000","last_undegraded":"2024-09-15T04:04:47.811226+0000","last_fullsized":"2024-09-15T04:04:47.811226+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:28:34.591535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036612400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":274,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.814025+0000","last_change":"2024-09-15T03:58:44.074233+0000","last_active":"2024-09-15T04:04:47.814025+0000","last_peered":"2024-09-15T04:04:47.814025+0000","last_clean":"2024-09-15T04:04:47.814025+0000","last_became_active":"2024-09-15T03:58:42.071652+0000","last_became_peered":"2024-09-15T03:58:42.071652+0000","last_unstale":"2024-09-15T04:04:47.814025+0000","last_undegraded":"2024-09-15T04:04:47.814025+0000","last_fullsized":"2024-09-15T04:04:47.814025+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:31:44.288660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030212199999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":369,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.811314+0000","last_change":"2024-09-15T03:58:38.057397+0000","last_active":"2024-09-15T04:04:47.811314+0000","last_peered":"2024-09-15T04:04:47.811314+0000","last_clean":"2024-09-15T04:04:47.811314+0000","last_became_active":"2024-09-15T03:58:38.056673+0000","last_became_peered":"2024-09-15T03:58:38.056673+0000","last_unstale":"2024-09-15T04:04:47.811314+0000","last_undegraded":"2024-09-15T04:04:47.811314+0000","last_fullsized":"2024-09-15T04:04:47.811314+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_clean_scrub_stamp":"2024-09-15T03:58:37.036429+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:38:58.792683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799916,"kb_used_data":92879700,"kb_used_omap":9352,"kb_used_meta":0,"kb_avail":92879700,"statfs":{"total":95927926784,"available":95108812800,"internally_reserved":0,"allocated":95108812800,"data_stored":95108812800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9577135,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48399999999999999,"15min":0.64000000000000001},"min":{"1min":0.32800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63800000000000001,"5min":0.70399999999999996,"15min":5.0810000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.50900000000000001,"15min":0.46100000000000002},"min":{"1min":0.33700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":0.51300000000000001}]},{"osd":1,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.54600000000000004,"15min":0.47899999999999998},"min":{"1min":0.26500000000000001,"5min":0.248,"15min":0.248},"max":{"1min":2.6720000000000002,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51700000000000002,"15min":0.46100000000000002},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66200000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.58199999999999996}]},{"osd":2,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.61799999999999999,"5min":0.63900000000000001,"15min":0.71099999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.51500000000000001,"15min":0.73799999999999999},"min":{"1min":0.38600000000000001,"5min":0.314,"15min":0.314},"max":{"1min":2.6320000000000001,"5min":2.6320000000000001,"15min":7.3520000000000003},"last":0.55600000000000005}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.24199999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.497,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.28100000000000003,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.57699999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.26500000000000001}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.38900000000000001,"15min":0.56999999999999995},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.5059999999999998,"5min":2.5059999999999998,"15min":5.1879999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.38200000000000001,"15min":0.57499999999999996},"min":{"1min":0.28399999999999997,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.51800000000000002,"5min":0.60699999999999998,"15min":5.1749999999999998},"last":0.307}]}]},{"osd":4,"up_from":11,"seq":47244640336,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799912,"kb_used_data":92879704,"kb_used_omap":9866,"kb_used_meta":0,"kb_avail":92879704,"statfs":{"total":95927926784,"available":95108816896,"internally_reserved":0,"allocated":95108816896,"data_stored":95108816896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10103386,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.78200000000000003},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.70299999999999996,"5min":1.3959999999999999,"15min":9.9429999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.51000000000000001,"15min":0.81699999999999995},"min":{"1min":0.26200000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.68500000000000005,"5min":1.272,"15min":11.509},"last":0.54500000000000004}]},{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56299999999999994,"15min":0.53700000000000003},"min":{"1min":0.311,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.76200000000000001,"5min":2.867,"15min":2.867},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56999999999999995,"15min":0.57699999999999996},"min":{"1min":0.33400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":1.7689999999999999,"5min":3.923,"15min":3.923},"last":0.47499999999999998}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51800000000000002,"15min":0.496},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69999999999999996,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.71799999999999997},"min":{"1min":0.35899999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.73999999999999999,"5min":1.3540000000000001,"15min":7.5140000000000002},"last":0.56299999999999994}]},{"osd":3,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.40000000000000002,"15min":0.39200000000000002},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.53300000000000003,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41999999999999998,"15min":0.42699999999999999},"min":{"1min":0.26600000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.55200000000000005,"5min":2.339,"15min":2.339},"last":0.48799999999999999}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44700000000000001,"15min":0.73599999999999999},"min":{"1min":0.33500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.5,"5min":3.7080000000000002,"15min":11.814},"last":0.505},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.438,"15min":0.42599999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.53300000000000003,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.42999999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640336,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":800500,"kb_used_data":92879116,"kb_used_omap":9496,"kb_used_meta":0,"kb_avail":92879116,"statfs":{"total":95927926784,"available":95108214784,"internally_reserved":0,"allocated":95108214784,"data_stored":95108214784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9724376,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.51600000000000001,"15min":0.48699999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.19900000000000001},"max":{"1min":0.63600000000000001,"5min":3.1429999999999998,"15min":3.1429999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48699999999999999,"15min":7.2050000000000001},"min":{"1min":0.33700000000000002,"5min":0.27900000000000003,"15min":0.23499999999999999},"max":{"1min":0.73699999999999999,"5min":0.76200000000000001,"15min":192.55600000000001},"last":0.35299999999999998}]},{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52100000000000002,"15min":7.2519999999999998},"min":{"1min":0.32300000000000001,"5min":0.30399999999999999,"15min":0.22600000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":192.50899999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55200000000000005,"15min":0.53200000000000003},"min":{"1min":0.32500000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.76700000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.44}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.501,"15min":0.64500000000000002},"min":{"1min":0.35399999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.624,"5min":0.79500000000000004,"15min":5.0419999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.73199999999999998},"min":{"1min":0.40100000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":0.77700000000000002,"15min":7.2930000000000001},"last":0.372}]},{"osd":4,"last update":"Sun Sep 15 04:04:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.36299999999999999,"15min":0.35499999999999998},"min":{"1min":0.223,"5min":0.222,"15min":0.222},"max":{"1min":0.46000000000000002,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.41299999999999998,"15min":0.38700000000000001},"min":{"1min":0.34499999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67300000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.45300000000000001}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39200000000000002},"min":{"1min":0.27100000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.625,"5min":0.67800000000000005,"15min":0.69999999999999996},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.39400000000000002,"15min":0.40400000000000003},"min":{"1min":0.30399999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.5,"5min":0.625,"15min":0.625},"last":0.42199999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799916,"kb_used_data":92879700,"kb_used_omap":9704,"kb_used_meta":0,"kb_avail":92879700,"statfs":{"total":95927926784,"available":95108812800,"internally_reserved":0,"allocated":95108812800,"data_stored":95108812800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9937685,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.316,"15min":0.34000000000000002},"min":{"1min":0.20699999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.47699999999999998,"5min":0.502,"15min":0.57699999999999996},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.33000000000000002,"15min":0.36399999999999999},"min":{"1min":0.223,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.44500000000000001,"5min":0.56999999999999995,"15min":0.65100000000000002},"last":0.21199999999999999}]},{"osd":1,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.34599999999999997},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48999999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.38,"5min":0.35899999999999999,"15min":0.372},"min":{"1min":0.28499999999999998,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54400000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.25900000000000001}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.67100000000000004,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72299999999999998,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.52200000000000002}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.66200000000000003,"5min":0.745,"15min":0.745},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54100000000000004,"15min":0.63500000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.754,"5min":1.208,"15min":3.1339999999999999},"last":0.502}]},{"osd":5,"last update":"Sun Sep 15 04:04:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.35999999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.65400000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73199999999999998,"5min":0.77800000000000002,"15min":2.9500000000000002},"last":0.54900000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640336,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799920,"kb_used_data":92879696,"kb_used_omap":9674,"kb_used_meta":0,"kb_avail":92879696,"statfs":{"total":95927926784,"available":95108808704,"internally_reserved":0,"allocated":95108808704,"data_stored":95108808704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9906960,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34499999999999997,"15min":0.34300000000000003},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.52700000000000002,"5min":1.101,"15min":1.101},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.35299999999999998,"15min":0.35099999999999998},"min":{"1min":0.26400000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.497,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.376,"15min":0.36699999999999999},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":0.55600000000000005,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.22},{"interface":"front","average":{"1min":0.374,"5min":0.36199999999999999,"15min":0.374},"min":{"1min":0.26300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.5,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.248}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52200000000000002,"15min":0.54800000000000004},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.247},"max":{"1min":0.65400000000000003,"5min":1.087,"15min":1.087},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.54900000000000004},"min":{"1min":0.35999999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68500000000000005,"5min":1.032,"15min":1.032},"last":0.47499999999999998}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.57699999999999996,"15min":0.56100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66400000000000003,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":0.495},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.623,"15min":0.59399999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68300000000000005,"5min":4.2930000000000001,"15min":4.2930000000000001},"last":0.51400000000000001}]},{"osd":5,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.33000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.72899999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.36899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70899999999999996,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.58299999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640336,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":800492,"kb_used_data":92879124,"kb_used_omap":10031,"kb_used_meta":0,"kb_avail":92879124,"statfs":{"total":95927926784,"available":95108222976,"internally_reserved":0,"allocated":95108222976,"data_stored":95108222976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10272647,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"min":{"1min":0.246,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.49099999999999999,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.35999999999999999},"min":{"1min":0.309,"5min":0.193,"15min":0.193},"max":{"1min":0.496,"5min":0.504,"15min":0.71199999999999997},"last":0.29699999999999999}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.34799999999999998,"15min":0.376},"min":{"1min":0.32400000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52000000000000002,"5min":0.67500000000000004,"15min":0.78300000000000003},"last":0.313},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.38800000000000001},"min":{"1min":0.29199999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.54100000000000004,"5min":0.65200000000000002,"15min":0.76000000000000001},"last":0.33700000000000002}]},{"osd":3,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.43099999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.80100000000000005,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72299999999999998,"5min":0.85599999999999998,"15min":0.88200000000000001},"last":0.57399999999999995}]},{"osd":4,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.46300000000000002,"5min":0.30099999999999999,"15min":0.29599999999999999},"max":{"1min":0.84599999999999997,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.57199999999999995,"15min":0.59599999999999997},"min":{"1min":0.53100000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.59599999999999997}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52800000000000002,"15min":0.54800000000000004},"min":{"1min":0.44500000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.56200000000000006,"15min":0.54800000000000004},"min":{"1min":0.47399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76000000000000001,"5min":1.044,"15min":1.044},"last":0.51800000000000002}]}]}],"pool_statfs":[]}} 2024-09-15T04:05:06.464 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T04:05:06.465 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T04:05:06.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.556+0000 7fd879ee8640 10 monclient: get_monmap_and_config 2024-09-15T04:05:06.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.556+0000 7fd879ee8640 10 monclient: build_initial_monmap 2024-09-15T04:05:06.557 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.556+0000 7fd879ee8640 10 monclient: monmap: 2024-09-15T04:05:06.557 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:06.557 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.557 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:06.557250+0000 2024-09-15T04:05:06.557 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:06.557250+0000 2024-09-15T04:05:06.557 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:06.557 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.557 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:06.557 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:06.558 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:06.558 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:06.558 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:06.558 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.556+0000 7fd879ee8640 1 Processor -- start 2024-09-15T04:05:06.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 1 -- start start 2024-09-15T04:05:06.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 10 monclient: init 2024-09-15T04:05:06.566 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T04:05:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 0x7fd87409c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 10 monclient(hunting): picked mon.noname-b con 0x7fd87409bf10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd87406ed10 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fd87406d050 0x7fd874135270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 10 monclient(hunting): picked mon.noname-c con 0x7fd87406d050 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87406c0a0 con 0x7fd87406ed10 2024-09-15T04:05:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87406ae00 con 0x7fd87406d050 2024-09-15T04:05:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87406a8f0 con 0x7fd87409bf10 2024-09-15T04:05:06.567 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd879ee8640 10 monclient(hunting): authenticate will time out at 1130.201517s 2024-09-15T04:05:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 0x7fd87409c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd872ffd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd87406ed10 0x7fd87409c820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58938/0 (socket says 172.21.15.110:58938) 2024-09-15T04:05:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 0x7fd87409c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35180/0 (socket says 172.21.15.110:35180) 2024-09-15T04:05:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd872ffd640 1 -- 172.21.15.110:0/3451679787 learned_addr learned my addr 172.21.15.110:0/3451679787 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 10 monclient(hunting): get_auth_request con 0x7fd87409bf10 auth_method 0 2024-09-15T04:05:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd873fff640 1 --2- 172.21.15.110:0/3451679787 >> v2:172.21.15.177:3300/0 conn(0x7fd87406d050 0x7fd874135270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd873fff640 10 monclient(hunting): get_auth_request con 0x7fd87406d050 auth_method 0 2024-09-15T04:05:06.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd873fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd873fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.569 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd873fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd873fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd873fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd873fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 10 monclient(hunting): handle_auth_done global_id 74150 payload 274 2024-09-15T04:05:06.570 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 1 -- 172.21.15.110:0/3451679787 >> v2:172.21.15.177:3300/0 conn(0x7fd87406d050 msgr2=0x7fd874135270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 1 --2- 172.21.15.110:0/3451679787 >> v2:172.21.15.177:3300/0 conn(0x7fd87406d050 0x7fd874135270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 1 -- 172.21.15.110:0/3451679787 >> v1:172.21.15.110:6789/0 conn(0x7fd87406ed10 legacy=0x7fd87409c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 10 monclient: _finish_hunting 0 2024-09-15T04:05:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 1 monclient: found mon.noname-b 2024-09-15T04:05:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:05:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.557+0000 7fd8737fe640 1 -- 172.21.15.110:0/3451679787 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87406bdc0 con 0x7fd87409bf10 2024-09-15T04:05:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd8737fe640 1 -- 172.21.15.110:0/3451679787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 msgr2=0x7fd87409c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd8737fe640 1 --2- 172.21.15.110:0/3451679787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 0x7fd87409c2e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd86400baf0 tx=0x7fd86400b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d49011432a8e1a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd8727fc640 1 -- 172.21.15.110:0/3451679787 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8640316c0 con 0x7fd87409bf10 2024-09-15T04:05:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd8727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd8727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:05:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd8727fc640 10 monclient: dump: 2024-09-15T04:05:06.571 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.572 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd8727fc640 10 monclient: _finish_auth 0 2024-09-15T04:05:06.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd8727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:06.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd8727fc640 1 -- 172.21.15.110:0/3451679787 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd864031820 con 0x7fd87409bf10 2024-09-15T04:05:06.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd879ee8640 5 monclient: authenticate success, global_id 74150 2024-09-15T04:05:06.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd879ee8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:06.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd8727fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:06.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd879ee8640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:06.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd8727fc640 1 -- 172.21.15.110:0/3451679787 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864031bc0 con 0x7fd87409bf10 2024-09-15T04:05:06.573 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd8727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd879ee8640 4 set_mon_vals no callback set 2024-09-15T04:05:06.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd8727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:06.574 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.560+0000 7fd8727fc640 10 monclient: dump: 2024-09-15T04:05:06.574 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.574 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.574 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.574 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.574 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.574 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.574 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.574 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.574 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.574 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.561+0000 7fd879ee8640 10 monclient: shutdown 2024-09-15T04:05:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.561+0000 7fd879ee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.561+0000 7fd879ee8640 1 -- 172.21.15.110:0/3451679787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 msgr2=0x7fd87409c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.561+0000 7fd879ee8640 1 --2- 172.21.15.110:0/3451679787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 0x7fd87409c2e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd86400baf0 tx=0x7fd86400b3b0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.561+0000 7fd879ee8640 1 -- 172.21.15.110:0/3451679787 shutdown_connections 2024-09-15T04:05:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.561+0000 7fd879ee8640 1 --2- 172.21.15.110:0/3451679787 >> v2:172.21.15.177:3300/0 conn(0x7fd87406d050 0x7fd874135270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.561+0000 7fd879ee8640 1 --2- 172.21.15.110:0/3451679787 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 0x7fd87409c2e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.561+0000 7fd879ee8640 1 -- 172.21.15.110:0/3451679787 >> 172.21.15.110:0/3451679787 conn(0x7fd87409d8f0 msgr2=0x7fd87409dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.561+0000 7fd879ee8640 1 -- 172.21.15.110:0/3451679787 shutdown_connections 2024-09-15T04:05:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.561+0000 7fd879ee8640 1 -- 172.21.15.110:0/3451679787 wait complete. 2024-09-15T04:05:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.561+0000 7fd879ee8640 10 monclient: build_initial_monmap 2024-09-15T04:05:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.561+0000 7fd879ee8640 10 monclient: monmap: 2024-09-15T04:05:06.575 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:06.576 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 1 Processor -- start 2024-09-15T04:05:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 1 -- start start 2024-09-15T04:05:06.576 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 10 monclient: init 2024-09-15T04:05:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T04:05:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd87406d050 0x7fd87413ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 10 monclient(hunting): picked mon.noname-b con 0x7fd87406d050 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87406ed10 0x7fd87412d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 10 monclient(hunting): picked mon.noname-a con 0x7fd87406ed10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 0x7fd8741170c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 10 monclient(hunting): picked mon.noname-c con 0x7fd87409bf10 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87406b2f0 con 0x7fd87406ed10 2024-09-15T04:05:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd874099ec0 con 0x7fd87406d050 2024-09-15T04:05:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.577 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87413d950 con 0x7fd87409bf10 2024-09-15T04:05:06.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:06.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd879ee8640 10 monclient(hunting): authenticate will time out at 1130.206895s 2024-09-15T04:05:06.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd873fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 0x7fd8741170c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.562+0000 7fd873fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 0x7fd8741170c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35188/0 (socket says 172.21.15.110:35188) 2024-09-15T04:05:06.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd872ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87406ed10 0x7fd87412d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 1 -- 172.21.15.110:0/3179421498 learned_addr learned my addr 172.21.15.110:0/3179421498 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:06.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 10 monclient(hunting): get_auth_request con 0x7fd87409bf10 auth_method 0 2024-09-15T04:05:06.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd872ffd640 10 monclient(hunting): get_auth_request con 0x7fd87406ed10 auth_method 0 2024-09-15T04:05:06.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd872ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.578 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd872ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd872ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd872ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd872ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd872ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 10 monclient(hunting): handle_auth_done global_id 74153 payload 824 2024-09-15T04:05:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd8737fe640 1 --2- 172.21.15.110:0/3179421498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd87406d050 0x7fd87413ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 1 -- 172.21.15.110:0/3179421498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd87406d050 msgr2=0x7fd87413ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 1 --2- 172.21.15.110:0/3179421498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd87406d050 0x7fd87413ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 1 -- 172.21.15.110:0/3179421498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87406ed10 msgr2=0x7fd87412d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.579 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 1 --2- 172.21.15.110:0/3179421498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87406ed10 0x7fd87412d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 10 monclient: _finish_hunting 0 2024-09-15T04:05:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 1 monclient: found mon.noname-c 2024-09-15T04:05:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.563+0000 7fd873fff640 1 -- 172.21.15.110:0/3179421498 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86400b630 con 0x7fd87409bf10 2024-09-15T04:05:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd873fff640 1 -- 172.21.15.110:0/3179421498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 msgr2=0x7fd8741170c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd873fff640 1 --2- 172.21.15.110:0/3179421498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 0x7fd8741170c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009ec0 tx=0x7fd86c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd86b7fe640 1 -- 172.21.15.110:0/3179421498 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c00ce10 con 0x7fd87409bf10 2024-09-15T04:05:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd86b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd86b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd86b7fe640 10 monclient: dump: 2024-09-15T04:05:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.580 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.581 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd86b7fe640 10 monclient: _finish_auth 0 2024-09-15T04:05:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd86b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd86b7fe640 1 -- 172.21.15.110:0/3179421498 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c00f040 con 0x7fd87409bf10 2024-09-15T04:05:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd879ee8640 5 monclient: authenticate success, global_id 74153 2024-09-15T04:05:06.581 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd879ee8640 10 monclient: _renew_subs 2024-09-15T04:05:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd879ee8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd879ee8640 1 -- 172.21.15.110:0/3179421498 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd87411a500 con 0x7fd87409bf10 2024-09-15T04:05:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd86b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd86b7fe640 1 -- 172.21.15.110:0/3179421498 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c0148e0 con 0x7fd87409bf10 2024-09-15T04:05:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd86b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd86b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd86b7fe640 10 monclient: dump: 2024-09-15T04:05:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.582 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.583 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.583 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.583 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.583 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd879ee8640 10 monclient: _renew_subs 2024-09-15T04:05:06.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd879ee8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:06.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd879ee8640 1 -- 172.21.15.110:0/3179421498 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87413cfb0 con 0x7fd87409bf10 2024-09-15T04:05:06.583 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.565+0000 7fd871ffb640 4 set_mon_vals no callback set 2024-09-15T04:05:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.566+0000 7fd86b7fe640 1 -- 172.21.15.110:0/3179421498 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fd86c014c90 con 0x7fd87409bf10 2024-09-15T04:05:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.566+0000 7fd879ee8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.566+0000 7fd879ee8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.566+0000 7fd879ee8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.566+0000 7fd879ee8640 1 -- 172.21.15.110:0/3179421498 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87411a690 con 0x7fd87409bf10 2024-09-15T04:05:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.566+0000 7fd86b7fe640 1 --2- 172.21.15.110:0/3179421498 >> v2:172.21.15.177:6812/28201 conn(0x7fd854038730 0x7fd85403abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.566+0000 7fd86b7fe640 1 -- 172.21.15.110:0/3179421498 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd86c04c560 con 0x7fd87409bf10 2024-09-15T04:05:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.566+0000 7fd8737fe640 1 --2- 172.21.15.110:0/3179421498 >> v2:172.21.15.177:6812/28201 conn(0x7fd854038730 0x7fd85403abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.566+0000 7fd8737fe640 10 monclient: get_auth_request con 0x7fd854038730 auth_method 0 2024-09-15T04:05:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.569+0000 7fd8737fe640 1 --2- 172.21.15.110:0/3179421498 >> v2:172.21.15.177:6812/28201 conn(0x7fd854038730 0x7fd85403abe0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd864036000 tx=0x7fd864037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:06.585 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.571+0000 7fd86b7fe640 1 -- 172.21.15.110:0/3179421498 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fd86c012940 con 0x7fd87409bf10 2024-09-15T04:05:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.571+0000 7fd86b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:06.586 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.571+0000 7fd86b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:06.722 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.721+0000 7fd879ee8640 1 -- 172.21.15.110:0/3179421498 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd874070950 con 0x7fd854038730 2024-09-15T04:05:06.724 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.724+0000 7fd86b7fe640 1 -- 172.21.15.110:0/3179421498 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+52166 (secure 0 0 0) 0x7fd874070950 con 0x7fd854038730 2024-09-15T04:05:06.725 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T04:05:06.725 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-09-15T04:05:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.726+0000 7fd879ee8640 1 -- 172.21.15.110:0/3179421498 >> v2:172.21.15.177:6812/28201 conn(0x7fd854038730 msgr2=0x7fd85403abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.727 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.726+0000 7fd879ee8640 1 --2- 172.21.15.110:0/3179421498 >> v2:172.21.15.177:6812/28201 conn(0x7fd854038730 0x7fd85403abe0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd864036000 tx=0x7fd864037000 comp rx=0 tx=0).stop 2024-09-15T04:05:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.726+0000 7fd879ee8640 10 monclient: shutdown 2024-09-15T04:05:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.726+0000 7fd879ee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.726+0000 7fd879ee8640 1 -- 172.21.15.110:0/3179421498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 msgr2=0x7fd8741170c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.726+0000 7fd879ee8640 1 --2- 172.21.15.110:0/3179421498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 0x7fd8741170c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009ec0 tx=0x7fd86c00b040 comp rx=0 tx=0).stop 2024-09-15T04:05:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.726+0000 7fd879ee8640 1 -- 172.21.15.110:0/3179421498 shutdown_connections 2024-09-15T04:05:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.726+0000 7fd879ee8640 1 --2- 172.21.15.110:0/3179421498 >> v2:172.21.15.177:6812/28201 conn(0x7fd854038730 0x7fd85403abe0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.726+0000 7fd879ee8640 1 --2- 172.21.15.110:0/3179421498 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fd87409bf10 0x7fd8741170c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.726+0000 7fd879ee8640 1 --2- 172.21.15.110:0/3179421498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd87406ed10 0x7fd87412d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.726+0000 7fd879ee8640 1 --2- 172.21.15.110:0/3179421498 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd87406d050 0x7fd87413ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.726+0000 7fd879ee8640 1 -- 172.21.15.110:0/3179421498 >> 172.21.15.110:0/3179421498 conn(0x7fd87409d8f0 msgr2=0x7fd87409f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.726+0000 7fd879ee8640 1 -- 172.21.15.110:0/3179421498 shutdown_connections 2024-09-15T04:05:06.728 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.726+0000 7fd879ee8640 1 -- 172.21.15.110:0/3179421498 wait complete. 2024-09-15T04:05:06.737 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-09-15T04:05:06.442175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562077696,"kb_used":4800656,"kb_used_data":557277040,"kb_used_omap":58127,"kb_used_meta":0,"kb_avail":557277040,"statfs":{"total":575567560704,"available":570651688960,"internally_reserved":0,"allocated":570651688960,"data_stored":570651688960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59522189,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6099,"num_objects":-50,"num_object_clones":0,"num_object_copies":-100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-50,"num_write_kb":-50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002544"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":281,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.812288+0000","last_change":"2024-09-15T03:58:44.072431+0000","last_active":"2024-09-15T04:04:47.812288+0000","last_peered":"2024-09-15T04:04:47.812288+0000","last_clean":"2024-09-15T04:04:47.812288+0000","last_became_active":"2024-09-15T03:58:42.157123+0000","last_became_peered":"2024-09-15T03:58:42.157123+0000","last_unstale":"2024-09-15T04:04:47.812288+0000","last_undegraded":"2024-09-15T04:04:47.812288+0000","last_fullsized":"2024-09-15T04:04:47.812288+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:46:37.797943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066518800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":302,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.808620+0000","last_change":"2024-09-15T03:58:44.072012+0000","last_active":"2024-09-15T04:04:47.808620+0000","last_peered":"2024-09-15T04:04:47.808620+0000","last_clean":"2024-09-15T04:04:47.808620+0000","last_became_active":"2024-09-15T03:58:42.069324+0000","last_became_peered":"2024-09-15T03:58:42.069324+0000","last_unstale":"2024-09-15T04:04:47.808620+0000","last_undegraded":"2024-09-15T04:04:47.808620+0000","last_fullsized":"2024-09-15T04:04:47.808620+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:44:04.033148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040535300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":281,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.811753+0000","last_change":"2024-09-15T03:58:44.072166+0000","last_active":"2024-09-15T04:04:47.811753+0000","last_peered":"2024-09-15T04:04:47.811753+0000","last_clean":"2024-09-15T04:04:47.811753+0000","last_became_active":"2024-09-15T03:58:42.155521+0000","last_became_peered":"2024-09-15T03:58:42.155521+0000","last_unstale":"2024-09-15T04:04:47.811753+0000","last_undegraded":"2024-09-15T04:04:47.811753+0000","last_fullsized":"2024-09-15T04:04:47.811753+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:42:01.920640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040527199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":302,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.808414+0000","last_change":"2024-09-15T03:58:44.072137+0000","last_active":"2024-09-15T04:04:47.808414+0000","last_peered":"2024-09-15T04:04:47.808414+0000","last_clean":"2024-09-15T04:04:47.808414+0000","last_became_active":"2024-09-15T03:58:42.155317+0000","last_became_peered":"2024-09-15T03:58:42.155317+0000","last_unstale":"2024-09-15T04:04:47.808414+0000","last_undegraded":"2024-09-15T04:04:47.808414+0000","last_fullsized":"2024-09-15T04:04:47.808414+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:55:28.163584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041321199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":275,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.813708+0000","last_change":"2024-09-15T03:58:44.074749+0000","last_active":"2024-09-15T04:04:47.813708+0000","last_peered":"2024-09-15T04:04:47.813708+0000","last_clean":"2024-09-15T04:04:47.813708+0000","last_became_active":"2024-09-15T03:58:42.071376+0000","last_became_peered":"2024-09-15T03:58:42.071376+0000","last_unstale":"2024-09-15T04:04:47.813708+0000","last_undegraded":"2024-09-15T04:04:47.813708+0000","last_fullsized":"2024-09-15T04:04:47.813708+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:42:52.640034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080084100000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":289,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.809641+0000","last_change":"2024-09-15T03:58:44.071453+0000","last_active":"2024-09-15T04:04:47.809641+0000","last_peered":"2024-09-15T04:04:47.809641+0000","last_clean":"2024-09-15T04:04:47.809641+0000","last_became_active":"2024-09-15T03:58:42.067012+0000","last_became_peered":"2024-09-15T03:58:42.067012+0000","last_unstale":"2024-09-15T04:04:47.809641+0000","last_undegraded":"2024-09-15T04:04:47.809641+0000","last_fullsized":"2024-09-15T04:04:47.809641+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:04:08.843151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040766700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":281,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.811226+0000","last_change":"2024-09-15T03:58:44.072155+0000","last_active":"2024-09-15T04:04:47.811226+0000","last_peered":"2024-09-15T04:04:47.811226+0000","last_clean":"2024-09-15T04:04:47.811226+0000","last_became_active":"2024-09-15T03:58:42.069300+0000","last_became_peered":"2024-09-15T03:58:42.069300+0000","last_unstale":"2024-09-15T04:04:47.811226+0000","last_undegraded":"2024-09-15T04:04:47.811226+0000","last_fullsized":"2024-09-15T04:04:47.811226+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:28:34.591535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036612400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":274,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.814025+0000","last_change":"2024-09-15T03:58:44.074233+0000","last_active":"2024-09-15T04:04:47.814025+0000","last_peered":"2024-09-15T04:04:47.814025+0000","last_clean":"2024-09-15T04:04:47.814025+0000","last_became_active":"2024-09-15T03:58:42.071652+0000","last_became_peered":"2024-09-15T03:58:42.071652+0000","last_unstale":"2024-09-15T04:04:47.814025+0000","last_undegraded":"2024-09-15T04:04:47.814025+0000","last_fullsized":"2024-09-15T04:04:47.814025+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:31:44.288660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030212199999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":369,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.811314+0000","last_change":"2024-09-15T03:58:38.057397+0000","last_active":"2024-09-15T04:04:47.811314+0000","last_peered":"2024-09-15T04:04:47.811314+0000","last_clean":"2024-09-15T04:04:47.811314+0000","last_became_active":"2024-09-15T03:58:38.056673+0000","last_became_peered":"2024-09-15T03:58:38.056673+0000","last_unstale":"2024-09-15T04:04:47.811314+0000","last_undegraded":"2024-09-15T04:04:47.811314+0000","last_fullsized":"2024-09-15T04:04:47.811314+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_clean_scrub_stamp":"2024-09-15T03:58:37.036429+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:38:58.792683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799916,"kb_used_data":92879700,"kb_used_omap":9352,"kb_used_meta":0,"kb_avail":92879700,"statfs":{"total":95927926784,"available":95108812800,"internally_reserved":0,"allocated":95108812800,"data_stored":95108812800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9577135,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48399999999999999,"15min":0.64000000000000001},"min":{"1min":0.32800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63800000000000001,"5min":0.70399999999999996,"15min":5.0810000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.50900000000000001,"15min":0.46100000000000002},"min":{"1min":0.33700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":0.51300000000000001}]},{"osd":1,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.54600000000000004,"15min":0.47899999999999998},"min":{"1min":0.26500000000000001,"5min":0.248,"15min":0.248},"max":{"1min":2.6720000000000002,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51700000000000002,"15min":0.46100000000000002},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66200000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.58199999999999996}]},{"osd":2,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.61799999999999999,"5min":0.63900000000000001,"15min":0.71099999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.51500000000000001,"15min":0.73799999999999999},"min":{"1min":0.38600000000000001,"5min":0.314,"15min":0.314},"max":{"1min":2.6320000000000001,"5min":2.6320000000000001,"15min":7.3520000000000003},"last":0.55600000000000005}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.24199999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.497,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.28100000000000003,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.57699999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.26500000000000001}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.38900000000000001,"15min":0.56999999999999995},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.5059999999999998,"5min":2.5059999999999998,"15min":5.1879999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.38200000000000001,"15min":0.57499999999999996},"min":{"1min":0.28399999999999997,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.51800000000000002,"5min":0.60699999999999998,"15min":5.1749999999999998},"last":0.307}]}]},{"osd":4,"up_from":11,"seq":47244640336,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799912,"kb_used_data":92879704,"kb_used_omap":9866,"kb_used_meta":0,"kb_avail":92879704,"statfs":{"total":95927926784,"available":95108816896,"internally_reserved":0,"allocated":95108816896,"data_stored":95108816896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10103386,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.78200000000000003},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.70299999999999996,"5min":1.3959999999999999,"15min":9.9429999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.51000000000000001,"15min":0.81699999999999995},"min":{"1min":0.26200000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.68500000000000005,"5min":1.272,"15min":11.509},"last":0.54500000000000004}]},{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56299999999999994,"15min":0.53700000000000003},"min":{"1min":0.311,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.76200000000000001,"5min":2.867,"15min":2.867},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56999999999999995,"15min":0.57699999999999996},"min":{"1min":0.33400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":1.7689999999999999,"5min":3.923,"15min":3.923},"last":0.47499999999999998}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51800000000000002,"15min":0.496},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69999999999999996,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.71799999999999997},"min":{"1min":0.35899999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.73999999999999999,"5min":1.3540000000000001,"15min":7.5140000000000002},"last":0.56299999999999994}]},{"osd":3,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.40000000000000002,"15min":0.39200000000000002},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.53300000000000003,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41999999999999998,"15min":0.42699999999999999},"min":{"1min":0.26600000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.55200000000000005,"5min":2.339,"15min":2.339},"last":0.48799999999999999}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44700000000000001,"15min":0.73599999999999999},"min":{"1min":0.33500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.5,"5min":3.7080000000000002,"15min":11.814},"last":0.505},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.438,"15min":0.42599999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.53300000000000003,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.42999999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640336,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":800500,"kb_used_data":92879116,"kb_used_omap":9496,"kb_used_meta":0,"kb_avail":92879116,"statfs":{"total":95927926784,"available":95108214784,"internally_reserved":0,"allocated":95108214784,"data_stored":95108214784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9724376,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.51600000000000001,"15min":0.48699999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.19900000000000001},"max":{"1min":0.63600000000000001,"5min":3.1429999999999998,"15min":3.1429999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48699999999999999,"15min":7.2050000000000001},"min":{"1min":0.33700000000000002,"5min":0.27900000000000003,"15min":0.23499999999999999},"max":{"1min":0.73699999999999999,"5min":0.76200000000000001,"15min":192.55600000000001},"last":0.35299999999999998}]},{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52100000000000002,"15min":7.2519999999999998},"min":{"1min":0.32300000000000001,"5min":0.30399999999999999,"15min":0.22600000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":192.50899999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55200000000000005,"15min":0.53200000000000003},"min":{"1min":0.32500000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.76700000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.44}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.501,"15min":0.64500000000000002},"min":{"1min":0.35399999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.624,"5min":0.79500000000000004,"15min":5.0419999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.73199999999999998},"min":{"1min":0.40100000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":0.77700000000000002,"15min":7.2930000000000001},"last":0.372}]},{"osd":4,"last update":"Sun Sep 15 04:04:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.36299999999999999,"15min":0.35499999999999998},"min":{"1min":0.223,"5min":0.222,"15min":0.222},"max":{"1min":0.46000000000000002,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.41299999999999998,"15min":0.38700000000000001},"min":{"1min":0.34499999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67300000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.45300000000000001}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39200000000000002},"min":{"1min":0.27100000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.625,"5min":0.67800000000000005,"15min":0.69999999999999996},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.39400000000000002,"15min":0.40400000000000003},"min":{"1min":0.30399999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.5,"5min":0.625,"15min":0.625},"last":0.42199999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799916,"kb_used_data":92879700,"kb_used_omap":9704,"kb_used_meta":0,"kb_avail":92879700,"statfs":{"total":95927926784,"available":95108812800,"internally_reserved":0,"allocated":95108812800,"data_stored":95108812800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9937685,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.316,"15min":0.34000000000000002},"min":{"1min":0.20699999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.47699999999999998,"5min":0.502,"15min":0.57699999999999996},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.33000000000000002,"15min":0.36399999999999999},"min":{"1min":0.223,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.44500000000000001,"5min":0.56999999999999995,"15min":0.65100000000000002},"last":0.21199999999999999}]},{"osd":1,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.34599999999999997},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48999999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.38,"5min":0.35899999999999999,"15min":0.372},"min":{"1min":0.28499999999999998,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54400000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.25900000000000001}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.67100000000000004,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72299999999999998,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.52200000000000002}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.66200000000000003,"5min":0.745,"15min":0.745},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54100000000000004,"15min":0.63500000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.754,"5min":1.208,"15min":3.1339999999999999},"last":0.502}]},{"osd":5,"last update":"Sun Sep 15 04:04:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.35999999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.65400000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73199999999999998,"5min":0.77800000000000002,"15min":2.9500000000000002},"last":0.54900000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640336,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799920,"kb_used_data":92879696,"kb_used_omap":9674,"kb_used_meta":0,"kb_avail":92879696,"statfs":{"total":95927926784,"available":95108808704,"internally_reserved":0,"allocated":95108808704,"data_stored":95108808704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9906960,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34499999999999997,"15min":0.34300000000000003},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.52700000000000002,"5min":1.101,"15min":1.101},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.35299999999999998,"15min":0.35099999999999998},"min":{"1min":0.26400000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.497,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.376,"15min":0.36699999999999999},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":0.55600000000000005,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.22},{"interface":"front","average":{"1min":0.374,"5min":0.36199999999999999,"15min":0.374},"min":{"1min":0.26300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.5,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.248}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52200000000000002,"15min":0.54800000000000004},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.247},"max":{"1min":0.65400000000000003,"5min":1.087,"15min":1.087},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.54900000000000004},"min":{"1min":0.35999999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68500000000000005,"5min":1.032,"15min":1.032},"last":0.47499999999999998}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.57699999999999996,"15min":0.56100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66400000000000003,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":0.495},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.623,"15min":0.59399999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68300000000000005,"5min":4.2930000000000001,"15min":4.2930000000000001},"last":0.51400000000000001}]},{"osd":5,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.33000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.72899999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.36899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70899999999999996,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.58299999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640336,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":800492,"kb_used_data":92879124,"kb_used_omap":10031,"kb_used_meta":0,"kb_avail":92879124,"statfs":{"total":95927926784,"available":95108222976,"internally_reserved":0,"allocated":95108222976,"data_stored":95108222976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10272647,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"min":{"1min":0.246,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.49099999999999999,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.35999999999999999},"min":{"1min":0.309,"5min":0.193,"15min":0.193},"max":{"1min":0.496,"5min":0.504,"15min":0.71199999999999997},"last":0.29699999999999999}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.34799999999999998,"15min":0.376},"min":{"1min":0.32400000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52000000000000002,"5min":0.67500000000000004,"15min":0.78300000000000003},"last":0.313},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.38800000000000001},"min":{"1min":0.29199999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.54100000000000004,"5min":0.65200000000000002,"15min":0.76000000000000001},"last":0.33700000000000002}]},{"osd":3,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.43099999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.80100000000000005,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72299999999999998,"5min":0.85599999999999998,"15min":0.88200000000000001},"last":0.57399999999999995}]},{"osd":4,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.46300000000000002,"5min":0.30099999999999999,"15min":0.29599999999999999},"max":{"1min":0.84599999999999997,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.57199999999999995,"15min":0.59599999999999997},"min":{"1min":0.53100000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.59599999999999997}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52800000000000002,"15min":0.54800000000000004},"min":{"1min":0.44500000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.56200000000000006,"15min":0.54800000000000004},"min":{"1min":0.47399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76000000000000001,"5min":1.044,"15min":1.044},"last":0.51800000000000002}]}]}],"pool_statfs":[]}} 2024-09-15T04:05:06.738 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T04:05:06.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.829+0000 7f115709f640 10 monclient: get_monmap_and_config 2024-09-15T04:05:06.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.829+0000 7f115709f640 10 monclient: build_initial_monmap 2024-09-15T04:05:06.830 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.829+0000 7f115709f640 10 monclient: monmap: 2024-09-15T04:05:06.830 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:06.830 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.830 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:06.830306+0000 2024-09-15T04:05:06.830 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:06.830306+0000 2024-09-15T04:05:06.830 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:06.830 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.830 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:06.831 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:06.831 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:06.831 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:06.831 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:06.831 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 1 Processor -- start 2024-09-15T04:05:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 1 -- start start 2024-09-15T04:05:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 10 monclient: init 2024-09-15T04:05:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-15T04:05:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1150131630 0x7f1150131a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 10 monclient(hunting): picked mon.noname-a con 0x7f1150131630 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1150131f40 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:06.833 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f115006ecb0 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f115009f320 con 0x7f1150131f40 2024-09-15T04:05:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1150068390 con 0x7f115006ecb0 2024-09-15T04:05:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1150132310 con 0x7f1150131630 2024-09-15T04:05:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f115709f640 10 monclient(hunting): authenticate will time out at 1130.474562s 2024-09-15T04:05:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1150131630 0x7f1150131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f114ffff640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1150131f40 0x7f115006e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58954/0 (socket says 172.21.15.110:58954) 2024-09-15T04:05:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f114ffff640 1 -- 172.21.15.110:0/3669477203 learned_addr learned my addr 172.21.15.110:0/3669477203 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:06.834 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 10 monclient(hunting): get_auth_request con 0x7f1150131630 auth_method 0 2024-09-15T04:05:06.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 10 monclient(hunting): handle_auth_done global_id 55206 payload 274 2024-09-15T04:05:06.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 1 -- 172.21.15.110:0/3669477203 >> v1:172.21.15.177:6789/0 conn(0x7f115006ecb0 legacy=0x7f1150138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:06.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 1 -- 172.21.15.110:0/3669477203 >> v1:172.21.15.110:6789/0 conn(0x7f1150131f40 legacy=0x7f115006e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:06.835 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 10 monclient: _finish_hunting 0 2024-09-15T04:05:06.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 1 monclient: found mon.noname-a 2024-09-15T04:05:06.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:06.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:06.836 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.830+0000 7f1154e3b640 1 -- 172.21.15.110:0/3669477203 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1150069570 con 0x7f1150131630 2024-09-15T04:05:06.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f1154e3b640 1 -- 172.21.15.110:0/3669477203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1150131630 msgr2=0x7f1150131a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:06.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f1154e3b640 1 --2- 172.21.15.110:0/3669477203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1150131630 0x7f1150131a00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1148009850 tx=0x7f11480090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7bbde2adf28916 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:06.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f114f7fe640 1 -- 172.21.15.110:0/3669477203 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148032440 con 0x7f1150131630 2024-09-15T04:05:06.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f114f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f114f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:06.842 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f114f7fe640 10 monclient: dump: 2024-09-15T04:05:06.842 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.842 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.843 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f114f7fe640 10 monclient: _finish_auth 0 2024-09-15T04:05:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f114f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f114f7fe640 1 -- 172.21.15.110:0/3669477203 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11480325a0 con 0x7f1150131630 2024-09-15T04:05:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f115709f640 5 monclient: authenticate success, global_id 55206 2024-09-15T04:05:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f115709f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:06.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f114f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:06.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f115709f640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:06.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f114f7fe640 1 -- 172.21.15.110:0/3669477203 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148032940 con 0x7f1150131630 2024-09-15T04:05:06.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f114f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f115709f640 4 set_mon_vals no callback set 2024-09-15T04:05:06.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f114f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:06.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.833+0000 7f114f7fe640 10 monclient: dump: 2024-09-15T04:05:06.844 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.844 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.844 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.844 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.844 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.844 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.844 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.844 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.845 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.845 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.834+0000 7f115709f640 10 monclient: shutdown 2024-09-15T04:05:06.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.834+0000 7f115709f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:06.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.834+0000 7f115709f640 1 -- 172.21.15.110:0/3669477203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1150131630 msgr2=0x7f1150131a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.834+0000 7f115709f640 1 --2- 172.21.15.110:0/3669477203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1150131630 0x7f1150131a00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1148009850 tx=0x7f11480090f0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.834+0000 7f115709f640 1 -- 172.21.15.110:0/3669477203 shutdown_connections 2024-09-15T04:05:06.845 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.834+0000 7f115709f640 1 --2- 172.21.15.110:0/3669477203 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1150131630 0x7f1150131a00 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.834+0000 7f115709f640 1 -- 172.21.15.110:0/3669477203 >> 172.21.15.110:0/3669477203 conn(0x7f11500639f0 msgr2=0x7f11500a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:06.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.834+0000 7f115709f640 1 -- 172.21.15.110:0/3669477203 shutdown_connections 2024-09-15T04:05:06.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.834+0000 7f115709f640 1 -- 172.21.15.110:0/3669477203 wait complete. 2024-09-15T04:05:06.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.834+0000 7f115709f640 10 monclient: build_initial_monmap 2024-09-15T04:05:06.848 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.834+0000 7f115709f640 10 monclient: monmap: 2024-09-15T04:05:06.848 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:06.849 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:06.849 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:06.849 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:06.849 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:06.849 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.849 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:06.849 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:06.849 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:06.849 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 1 Processor -- start 2024-09-15T04:05:06.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 1 -- start start 2024-09-15T04:05:06.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 10 monclient: init 2024-09-15T04:05:06.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:06.849 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T04:05:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f115006ecb0 0x7f1150138160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 10 monclient(hunting): picked mon.noname-b con 0x7f115006ecb0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1150131630 0x7f1150133e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 10 monclient(hunting): picked mon.noname-c con 0x7f1150131630 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1150131f40 0x7f11501343c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 10 monclient(hunting): picked mon.noname-a con 0x7f1150131f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11500688f0 con 0x7f1150131f40 2024-09-15T04:05:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1150131060 con 0x7f115006ecb0 2024-09-15T04:05:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1150138720 con 0x7f1150131630 2024-09-15T04:05:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:06.850 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115709f640 10 monclient(hunting): authenticate will time out at 1130.479908s 2024-09-15T04:05:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f114ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1150131630 0x7f1150133e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.835+0000 7f115563c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1150131f40 0x7f11501343c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f114ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1150131630 0x7f1150133e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35202/0 (socket says 172.21.15.110:35202) 2024-09-15T04:05:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1150131f40 0x7f11501343c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42846/0 (socket says 172.21.15.110:42846) 2024-09-15T04:05:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f114ffff640 1 -- 172.21.15.110:0/4232490764 learned_addr learned my addr 172.21.15.110:0/4232490764 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f114ffff640 10 monclient(hunting): get_auth_request con 0x7f1150131630 auth_method 0 2024-09-15T04:05:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f114ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f114ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f114ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 10 monclient(hunting): get_auth_request con 0x7f1150131f40 auth_method 0 2024-09-15T04:05:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.851 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f114ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f1154e3b640 1 --2- 172.21.15.110:0/4232490764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f115006ecb0 0x7f1150138160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f1154e3b640 10 monclient(hunting): get_auth_request con 0x7f115006ecb0 auth_method 0 2024-09-15T04:05:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f1154e3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f1154e3b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f1154e3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 10 monclient(hunting): handle_auth_done global_id 55209 payload 824 2024-09-15T04:05:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 1 -- 172.21.15.110:0/4232490764 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1150131630 msgr2=0x7f1150133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 1 --2- 172.21.15.110:0/4232490764 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1150131630 0x7f1150133e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.852 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 1 -- 172.21.15.110:0/4232490764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f115006ecb0 msgr2=0x7f1150138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:06.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 1 --2- 172.21.15.110:0/4232490764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f115006ecb0 0x7f1150138160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:06.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 10 monclient: _finish_hunting 0 2024-09-15T04:05:06.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 1 monclient: found mon.noname-a 2024-09-15T04:05:06.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:06.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:06.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f115563c640 1 -- 172.21.15.110:0/4232490764 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1148009390 con 0x7f1150131f40 2024-09-15T04:05:06.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.836+0000 7f114ffff640 1 --2- 172.21.15.110:0/4232490764 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1150131630 0x7f1150133e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:06.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f115563c640 1 -- 172.21.15.110:0/4232490764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1150131f40 msgr2=0x7f11501343c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:06.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f115563c640 1 --2- 172.21.15.110:0/4232490764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1150131f40 0x7f11501343c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f113800b490 tx=0x7f1138006f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:06.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f114cff9640 1 -- 172.21.15.110:0/4232490764 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113800f720 con 0x7f1150131f40 2024-09-15T04:05:06.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f114cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f114cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:06.853 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f114cff9640 10 monclient: dump: 2024-09-15T04:05:06.853 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.854 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f114cff9640 10 monclient: _finish_auth 0 2024-09-15T04:05:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f114cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f114cff9640 1 -- 172.21.15.110:0/4232490764 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113800f8a0 con 0x7f1150131f40 2024-09-15T04:05:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f115709f640 5 monclient: authenticate success, global_id 55209 2024-09-15T04:05:06.854 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f114cff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:06.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f114cff9640 1 -- 172.21.15.110:0/4232490764 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f113800fc60 con 0x7f1150131f40 2024-09-15T04:05:06.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f115709f640 10 monclient: _renew_subs 2024-09-15T04:05:06.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f115709f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:06.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f115709f640 1 -- 172.21.15.110:0/4232490764 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1150137780 con 0x7f1150131f40 2024-09-15T04:05:06.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f114cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:06.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f114cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:06.855 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f114cff9640 10 monclient: dump: 2024-09-15T04:05:06.855 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:06.855 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:06.855 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.855 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:06.855 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:06.855 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:06.855 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:06.856 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:06.856 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:06.856 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:06.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f114effd640 4 set_mon_vals no callback set 2024-09-15T04:05:06.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f115709f640 10 monclient: _renew_subs 2024-09-15T04:05:06.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f115709f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:06.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.838+0000 7f115709f640 1 -- 172.21.15.110:0/4232490764 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1150137000 con 0x7f1150131f40 2024-09-15T04:05:06.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.839+0000 7f114cff9640 1 -- 172.21.15.110:0/4232490764 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f1138011610 con 0x7f1150131f40 2024-09-15T04:05:06.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.839+0000 7f115709f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:06.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.839+0000 7f115709f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:06.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.839+0000 7f115709f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:06.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.839+0000 7f115709f640 1 -- 172.21.15.110:0/4232490764 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1150137c30 con 0x7f1150131f40 2024-09-15T04:05:06.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.839+0000 7f114cff9640 1 --2- 172.21.15.110:0/4232490764 >> v2:172.21.15.177:6812/28201 conn(0x7f1130038700 0x7f113003abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:06.856 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.839+0000 7f114cff9640 1 -- 172.21.15.110:0/4232490764 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f113804a9a0 con 0x7f1150131f40 2024-09-15T04:05:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.840+0000 7f1154e3b640 1 --2- 172.21.15.110:0/4232490764 >> v2:172.21.15.177:6812/28201 conn(0x7f1130038700 0x7f113003abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.840+0000 7f1154e3b640 10 monclient: get_auth_request con 0x7f1130038700 auth_method 0 2024-09-15T04:05:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.842+0000 7f1154e3b640 1 --2- 172.21.15.110:0/4232490764 >> v2:172.21.15.177:6812/28201 conn(0x7f1130038700 0x7f113003abb0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1150069d10 tx=0x7f1148032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.844+0000 7f114cff9640 1 -- 172.21.15.110:0/4232490764 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f1150137c30 con 0x7f1150131f40 2024-09-15T04:05:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.844+0000 7f114cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:06.857 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.844+0000 7f114cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:06.998 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.998+0000 7f115709f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T04:05:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.998+0000 7f115709f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T04:05:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.998+0000 7f115709f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.998+0000 7f115709f640 1 -- 172.21.15.110:0/4232490764 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1150137c30 con 0x7f1150131f40 2024-09-15T04:05:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.998+0000 7f114cff9640 1 -- 172.21.15.110:0/4232490764 <== mon.0 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v315) v1 ==== 74+0+10033 (secure 0 0 0) 0x7f1150137c30 con 0x7f1150131f40 2024-09-15T04:05:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.998+0000 7f114cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T04:05:06.999 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:06.998+0000 7f114cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:05:07.000 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T04:05:07.000 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":315,"fsid":"802f2aac-d77f-438d-80e9-5e0c23519086","created":"2024-09-15T03:58:28.265907+0000","modified":"2024-09-15T04:04:48.807362+0000","last_up_change":"2024-09-15T03:58:35.032799+0000","last_in_change":"2024-09-15T03:58:31.538597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":100,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T03:58:36.398831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T03:58:41.004777+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"105a3b3d-e410-47fb-ac54-6a5d5da4a46f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":28367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":28367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":28367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":28367}]},"public_addr":"172.21.15.110:6808/28367","cluster_addr":"172.21.15.110:6809/28367","heartbeat_back_addr":"172.21.15.110:6811/28367","heartbeat_front_addr":"172.21.15.110:6810/28367","state":["exists","up"]},{"osd":1,"uuid":"77f0f205-6c64-495a-8edd-043de180efa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":28369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":28369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":28369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":28369}]},"public_addr":"172.21.15.110:6800/28369","cluster_addr":"172.21.15.110:6801/28369","heartbeat_back_addr":"172.21.15.110:6803/28369","heartbeat_front_addr":"172.21.15.110:6802/28369","state":["exists","up"]},{"osd":2,"uuid":"e8a42031-303e-4ad0-9896-e5b84cc95cfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":28368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":28368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":28368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":28368}]},"public_addr":"172.21.15.110:6804/28368","cluster_addr":"172.21.15.110:6805/28368","heartbeat_back_addr":"172.21.15.110:6807/28368","heartbeat_front_addr":"172.21.15.110:6806/28368","state":["exists","up"]},{"osd":3,"uuid":"0d65db82-4a0e-4f31-a10a-dcd71dcb269f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":28628}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6805","nonce":28628}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6807","nonce":28628}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":28628}]},"public_addr":"172.21.15.177:6804/28628","cluster_addr":"172.21.15.177:6805/28628","heartbeat_back_addr":"172.21.15.177:6807/28628","heartbeat_front_addr":"172.21.15.177:6806/28628","state":["exists","up"]},{"osd":4,"uuid":"e95e4f77-578d-4d03-bb4d-ee72b43bc227","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":28629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6801","nonce":28629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6803","nonce":28629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":28629}]},"public_addr":"172.21.15.177:6800/28629","cluster_addr":"172.21.15.177:6801/28629","heartbeat_back_addr":"172.21.15.177:6803/28629","heartbeat_front_addr":"172.21.15.177:6802/28629","state":["exists","up"]},{"osd":5,"uuid":"f16ffee6-1e25-432c-b6c2-909a72ce24a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":28630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6809","nonce":28630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6811","nonce":28630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":28630}]},"public_addr":"172.21.15.177:6808/28630","cluster_addr":"172.21.15.177:6809/28630","heartbeat_back_addr":"172.21.15.177:6811/28630","heartbeat_front_addr":"172.21.15.177:6810/28630","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.117560+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.137736+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.130715+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.173663+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.130009+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:58:33.172505+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T04:05:07.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:07.000+0000 7f115709f640 1 -- 172.21.15.110:0/4232490764 >> v2:172.21.15.177:6812/28201 conn(0x7f1130038700 msgr2=0x7f113003abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:07.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:07.000+0000 7f115709f640 1 --2- 172.21.15.110:0/4232490764 >> v2:172.21.15.177:6812/28201 conn(0x7f1130038700 0x7f113003abb0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1150069d10 tx=0x7f1148032000 comp rx=0 tx=0).stop 2024-09-15T04:05:07.001 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:07.000+0000 7f115709f640 10 monclient: shutdown 2024-09-15T04:05:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:07.000+0000 7f115709f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:07.000+0000 7f115709f640 1 -- 172.21.15.110:0/4232490764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1150131f40 msgr2=0x7f11501343c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:07.000+0000 7f115709f640 1 --2- 172.21.15.110:0/4232490764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1150131f40 0x7f11501343c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f113800b490 tx=0x7f1138006f50 comp rx=0 tx=0).stop 2024-09-15T04:05:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:07.000+0000 7f115709f640 1 -- 172.21.15.110:0/4232490764 shutdown_connections 2024-09-15T04:05:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:07.000+0000 7f115709f640 1 --2- 172.21.15.110:0/4232490764 >> v2:172.21.15.177:6812/28201 conn(0x7f1130038700 0x7f113003abb0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:07.000+0000 7f115709f640 1 --2- 172.21.15.110:0/4232490764 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1150131f40 0x7f11501343c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:07.000+0000 7f115709f640 1 --2- 172.21.15.110:0/4232490764 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f1150131630 0x7f1150133e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:07.000+0000 7f115709f640 1 --2- 172.21.15.110:0/4232490764 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f115006ecb0 0x7f1150138160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:07.000+0000 7f115709f640 1 -- 172.21.15.110:0/4232490764 >> 172.21.15.110:0/4232490764 conn(0x7f11500639f0 msgr2=0x7f11500a1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:07.000+0000 7f115709f640 1 -- 172.21.15.110:0/4232490764 shutdown_connections 2024-09-15T04:05:07.002 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:07.000+0000 7f115709f640 1 -- 172.21.15.110:0/4232490764 wait complete. 2024-09-15T04:05:08.012 INFO:tasks.ceph:Scrubbing osd.0 2024-09-15T04:05:08.013 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-15T04:05:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.112+0000 7f74762d4640 10 monclient: get_monmap_and_config 2024-09-15T04:05:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.112+0000 7f74762d4640 10 monclient: build_initial_monmap 2024-09-15T04:05:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.112+0000 7f74762d4640 10 monclient: monmap: 2024-09-15T04:05:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:08.113 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:08.113605+0000 2024-09-15T04:05:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:08.113605+0000 2024-09-15T04:05:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:08.114 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:08.114 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 1 Processor -- start 2024-09-15T04:05:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 1 -- start start 2024-09-15T04:05:08.116 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 10 monclient: init 2024-09-15T04:05:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-15T04:05:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f7470131630 0x7f7470131a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 10 monclient(hunting): picked mon.noname-c con 0x7f7470131630 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7470131f40 0x7f747006e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 10 monclient(hunting): picked mon.noname-a con 0x7f7470131f40 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f747006eae0 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747009f320 con 0x7f747006eae0 2024-09-15T04:05:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7470132340 con 0x7f7470131f40 2024-09-15T04:05:08.117 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7470068390 con 0x7f7470131630 2024-09-15T04:05:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f74762d4640 10 monclient(hunting): authenticate will time out at 1131.757866s 2024-09-15T04:05:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f746f7fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7470131f40 0x7f747006e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.113+0000 7f7474871640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f747006eae0 0x7f747009d560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58958/0 (socket says 172.21.15.110:58958) 2024-09-15T04:05:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f7474871640 1 -- 172.21.15.110:0/3028666308 learned_addr learned my addr 172.21.15.110:0/3028666308 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 10 monclient(hunting): get_auth_request con 0x7f7470131f40 auth_method 0 2024-09-15T04:05:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.118 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746ffff640 1 --2- 172.21.15.110:0/3028666308 >> v2:172.21.15.177:3300/0 conn(0x7f7470131630 0x7f7470131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746ffff640 10 monclient(hunting): get_auth_request con 0x7f7470131630 auth_method 0 2024-09-15T04:05:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 10 monclient(hunting): handle_auth_done global_id 55212 payload 274 2024-09-15T04:05:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 1 -- 172.21.15.110:0/3028666308 >> v2:172.21.15.177:3300/0 conn(0x7f7470131630 msgr2=0x7f7470131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 1 --2- 172.21.15.110:0/3028666308 >> v2:172.21.15.177:3300/0 conn(0x7f7470131630 0x7f7470131a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 1 -- 172.21.15.110:0/3028666308 >> v1:172.21.15.110:6789/0 conn(0x7f747006eae0 legacy=0x7f747009d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 10 monclient: _finish_hunting 0 2024-09-15T04:05:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 1 monclient: found mon.noname-a 2024-09-15T04:05:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:08.119 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746f7fe640 1 -- 172.21.15.110:0/3028666308 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7470069570 con 0x7f7470131f40 2024-09-15T04:05:08.120 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.114+0000 7f746ffff640 1 --2- 172.21.15.110:0/3028666308 >> v2:172.21.15.177:3300/0 conn(0x7f7470131630 0x7f7470131a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T04:05:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f746f7fe640 1 -- 172.21.15.110:0/3028666308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7470131f40 msgr2=0x7f747006e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f746f7fe640 1 --2- 172.21.15.110:0/3028666308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7470131f40 0x7f747006e5a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f747006bed0 tx=0x7f74640090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937cacee6e8a6b45 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f746effd640 1 -- 172.21.15.110:0/3028666308 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746402ae90 con 0x7f7470131f40 2024-09-15T04:05:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f746effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.126 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f746effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f746effd640 10 monclient: dump: 2024-09-15T04:05:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.127 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f746effd640 10 monclient: _finish_auth 0 2024-09-15T04:05:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f746effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:08.127 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f746effd640 1 -- 172.21.15.110:0/3028666308 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7464032440 con 0x7f7470131f40 2024-09-15T04:05:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f74762d4640 5 monclient: authenticate success, global_id 55212 2024-09-15T04:05:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f74762d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f746effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f74762d4640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f746effd640 1 -- 172.21.15.110:0/3028666308 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7464032820 con 0x7f7470131f40 2024-09-15T04:05:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f746effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f74762d4640 4 set_mon_vals no callback set 2024-09-15T04:05:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f746effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f746effd640 10 monclient: dump: 2024-09-15T04:05:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.128 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.129 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f74762d4640 10 monclient: shutdown 2024-09-15T04:05:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f74762d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f74762d4640 1 -- 172.21.15.110:0/3028666308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7470131f40 msgr2=0x7f747006e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f74762d4640 1 --2- 172.21.15.110:0/3028666308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7470131f40 0x7f747006e5a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f747006bed0 tx=0x7f74640090f0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f74762d4640 1 -- 172.21.15.110:0/3028666308 shutdown_connections 2024-09-15T04:05:08.129 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f74762d4640 1 --2- 172.21.15.110:0/3028666308 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7470131f40 0x7f747006e5a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f74762d4640 1 --2- 172.21.15.110:0/3028666308 >> v2:172.21.15.177:3300/0 conn(0x7f7470131630 0x7f7470131a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f74762d4640 1 -- 172.21.15.110:0/3028666308 >> 172.21.15.110:0/3028666308 conn(0x7f74700639f0 msgr2=0x7f74700a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f74762d4640 1 -- 172.21.15.110:0/3028666308 shutdown_connections 2024-09-15T04:05:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.117+0000 7f74762d4640 1 -- 172.21.15.110:0/3028666308 wait complete. 2024-09-15T04:05:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.118+0000 7f74762d4640 10 monclient: build_initial_monmap 2024-09-15T04:05:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.118+0000 7f74762d4640 10 monclient: monmap: 2024-09-15T04:05:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:08.130 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.118+0000 7f74762d4640 1 Processor -- start 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.118+0000 7f74762d4640 1 -- start start 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.118+0000 7f74762d4640 10 monclient: init 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f747006eae0 0x7f747009bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 10 monclient(hunting): picked mon.noname-b con 0x7f747006eae0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7470131630 0x7f747012e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 10 monclient(hunting): picked mon.noname-a con 0x7f7470131630 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7470131f40 0x7f74701191c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 10 monclient(hunting): picked mon.noname-c con 0x7f7470131f40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.131 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747011bcf0 con 0x7f7470131630 2024-09-15T04:05:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74700688c0 con 0x7f747006eae0 2024-09-15T04:05:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74700a2670 con 0x7f7470131f40 2024-09-15T04:05:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f74762d4640 10 monclient(hunting): authenticate will time out at 1131.763354s 2024-09-15T04:05:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7470131630 0x7f747012e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7470131630 0x7f747012e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42868/0 (socket says 172.21.15.110:42868) 2024-09-15T04:05:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 1 -- 172.21.15.110:0/452616058 learned_addr learned my addr 172.21.15.110:0/452616058 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 10 monclient(hunting): get_auth_request con 0x7f7470131630 auth_method 0 2024-09-15T04:05:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.132 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f7474871640 1 --2- 172.21.15.110:0/452616058 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7470131f40 0x7f74701191c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f7474871640 10 monclient(hunting): get_auth_request con 0x7f7470131f40 auth_method 0 2024-09-15T04:05:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f7474871640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f7474871640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f7474871640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f7474871640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f7474871640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f7474871640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746ffff640 1 --2- 172.21.15.110:0/452616058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f747006eae0 0x7f747009bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.133 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 10 monclient(hunting): handle_auth_done global_id 55215 payload 824 2024-09-15T04:05:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 1 -- 172.21.15.110:0/452616058 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7470131f40 msgr2=0x7f74701191c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 1 --2- 172.21.15.110:0/452616058 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f7470131f40 0x7f74701191c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 1 -- 172.21.15.110:0/452616058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f747006eae0 msgr2=0x7f747009bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 1 --2- 172.21.15.110:0/452616058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f747006eae0 0x7f747009bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 10 monclient: _finish_hunting 0 2024-09-15T04:05:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 1 monclient: found mon.noname-a 2024-09-15T04:05:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746f7fe640 1 -- 172.21.15.110:0/452616058 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7464009390 con 0x7f7470131630 2024-09-15T04:05:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746ffff640 10 monclient: get_auth_request con 0x7f747006eae0 auth_method 0 2024-09-15T04:05:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.119+0000 7f746ffff640 1 --2- 172.21.15.110:0/452616058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f747006eae0 0x7f747009bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:05:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.121+0000 7f746f7fe640 1 -- 172.21.15.110:0/452616058 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7470131630 msgr2=0x7f747012e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.121+0000 7f746f7fe640 1 --2- 172.21.15.110:0/452616058 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7470131630 0x7f747012e7f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7464009360 tx=0x7f7464032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f7457fff640 1 -- 172.21.15.110:0/452616058 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7464031c70 con 0x7f7470131630 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f7457fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f7457fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f7457fff640 10 monclient: dump: 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f7457fff640 10 monclient: _finish_auth 0 2024-09-15T04:05:08.135 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f7457fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f74762d4640 5 monclient: authenticate success, global_id 55215 2024-09-15T04:05:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f7457fff640 1 -- 172.21.15.110:0/452616058 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746402ae90 con 0x7f7470131630 2024-09-15T04:05:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f7457fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f7457fff640 1 -- 172.21.15.110:0/452616058 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746403d440 con 0x7f7470131630 2024-09-15T04:05:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f7457fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f7457fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f7457fff640 10 monclient: dump: 2024-09-15T04:05:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.136 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.137 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f74762d4640 10 monclient: _renew_subs 2024-09-15T04:05:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f74762d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f74762d4640 1 -- 172.21.15.110:0/452616058 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7470141b30 con 0x7f7470131630 2024-09-15T04:05:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f746e7fc640 4 set_mon_vals no callback set 2024-09-15T04:05:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f74762d4640 10 monclient: _renew_subs 2024-09-15T04:05:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f74762d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f74762d4640 1 -- 172.21.15.110:0/452616058 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747011b910 con 0x7f7470131630 2024-09-15T04:05:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f74762d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.122+0000 7f74762d4640 1 -- 172.21.15.110:0/452616058 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7470070950 con 0x7f7470131630 2024-09-15T04:05:08.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.123+0000 7f7457fff640 1 -- 172.21.15.110:0/452616058 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f7464039040 con 0x7f7470131630 2024-09-15T04:05:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.123+0000 7f7457fff640 1 --2- 172.21.15.110:0/452616058 >> v2:172.21.15.177:6812/28201 conn(0x7f745c038700 0x7f745c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.123+0000 7f7457fff640 1 -- 172.21.15.110:0/452616058 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f746406b430 con 0x7f7470131630 2024-09-15T04:05:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.123+0000 7f7457fff640 1 --2- 172.21.15.110:0/452616058 >> v2:172.21.15.110:6808/28367 conn(0x7f745c03e220 0x7f745c040630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.123+0000 7f7457fff640 1 -- 172.21.15.110:0/452616058 --> v2:172.21.15.110:6808/28367 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f745c040ca0 con 0x7f745c03e220 2024-09-15T04:05:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.123+0000 7f7457fff640 1 -- 172.21.15.110:0/452616058 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=315) v2 ==== 24+0+0 (secure 0 0 0) 0x7f746406d960 con 0x7f7470131630 2024-09-15T04:05:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.123+0000 7f7457fff640 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-15T04:05:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.123+0000 7f7474871640 1 --2- 172.21.15.110:0/452616058 >> v2:172.21.15.110:6808/28367 conn(0x7f745c03e220 0x7f745c040630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.123+0000 7f7474871640 10 monclient: get_auth_request con 0x7f745c03e220 auth_method 0 2024-09-15T04:05:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.123+0000 7f746ffff640 1 --2- 172.21.15.110:0/452616058 >> v2:172.21.15.177:6812/28201 conn(0x7f745c038700 0x7f745c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.123+0000 7f746ffff640 10 monclient: get_auth_request con 0x7f745c038700 auth_method 0 2024-09-15T04:05:08.138 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.126+0000 7f746ffff640 1 --2- 172.21.15.110:0/452616058 >> v2:172.21.15.177:6812/28201 conn(0x7f745c038700 0x7f745c03abb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7460006e70 tx=0x7f7460008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.127+0000 7f7474871640 1 --2- 172.21.15.110:0/452616058 >> v2:172.21.15.110:6808/28367 conn(0x7f745c03e220 0x7f745c040630 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.128+0000 7f7457fff640 1 -- 172.21.15.110:0/452616058 <== osd.0 v2:172.21.15.110:6808/28367 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f745c040ca0 con 0x7f745c03e220 2024-09-15T04:05:08.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.138+0000 7f74762d4640 1 -- 172.21.15.110:0/452616058 --> v2:172.21.15.110:6808/28367 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7470141b30 con 0x7f745c03e220 2024-09-15T04:05:08.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.138+0000 7f7457fff640 1 -- 172.21.15.110:0/452616058 <== osd.0 v2:172.21.15.110:6808/28367 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f7470141b30 con 0x7f745c03e220 2024-09-15T04:05:08.140 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:05:08.140 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T04:05:08.140 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:05:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.139+0000 7f74762d4640 1 -- 172.21.15.110:0/452616058 >> v2:172.21.15.110:6808/28367 conn(0x7f745c03e220 msgr2=0x7f745c040630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.139+0000 7f74762d4640 1 --2- 172.21.15.110:0/452616058 >> v2:172.21.15.110:6808/28367 conn(0x7f745c03e220 0x7f745c040630 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.139+0000 7f74762d4640 1 -- 172.21.15.110:0/452616058 >> v2:172.21.15.177:6812/28201 conn(0x7f745c038700 msgr2=0x7f745c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.139+0000 7f74762d4640 1 --2- 172.21.15.110:0/452616058 >> v2:172.21.15.177:6812/28201 conn(0x7f745c038700 0x7f745c03abb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7460006e70 tx=0x7f7460008040 comp rx=0 tx=0).stop 2024-09-15T04:05:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.139+0000 7f74762d4640 10 monclient: shutdown 2024-09-15T04:05:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.139+0000 7f74762d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.139+0000 7f74762d4640 1 -- 172.21.15.110:0/452616058 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7470131630 msgr2=0x7f747012e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.139+0000 7f74762d4640 1 --2- 172.21.15.110:0/452616058 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7470131630 0x7f747012e7f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7464009360 tx=0x7f7464032000 comp rx=0 tx=0).stop 2024-09-15T04:05:08.141 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.139+0000 7f7474871640 1 -- 172.21.15.110:0/452616058 reap_dead start 2024-09-15T04:05:08.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.139+0000 7f74762d4640 1 -- 172.21.15.110:0/452616058 shutdown_connections 2024-09-15T04:05:08.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.139+0000 7f74762d4640 1 -- 172.21.15.110:0/452616058 >> 172.21.15.110:0/452616058 conn(0x7f74700639f0 msgr2=0x7f74700a12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:08.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.139+0000 7f74762d4640 1 -- 172.21.15.110:0/452616058 shutdown_connections 2024-09-15T04:05:08.142 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.139+0000 7f74762d4640 1 -- 172.21.15.110:0/452616058 wait complete. 2024-09-15T04:05:08.149 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-15T04:05:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.242+0000 7fb6f4b00640 10 monclient: get_monmap_and_config 2024-09-15T04:05:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.242+0000 7fb6f4b00640 10 monclient: build_initial_monmap 2024-09-15T04:05:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.242+0000 7fb6f4b00640 10 monclient: monmap: 2024-09-15T04:05:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.243 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:08.243135+0000 2024-09-15T04:05:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:08.243135+0000 2024-09-15T04:05:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:08.244 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.242+0000 7fb6f4b00640 1 Processor -- start 2024-09-15T04:05:08.244 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.242+0000 7fb6f4b00640 1 -- start start 2024-09-15T04:05:08.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.242+0000 7fb6f4b00640 10 monclient: init 2024-09-15T04:05:08.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6f4b00640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:08.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6f4b00640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-15T04:05:08.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6f4b00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6f00a1a10 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:08.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6f4b00640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7fb6f009ee80 0x7fb6f0131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6f4b00640 10 monclient(hunting): picked mon.noname-c con 0x7fb6f009ee80 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:08.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6f4b00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6f01369e0 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:08.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6f4b00640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f009f2c0 con 0x7fb6f01369e0 2024-09-15T04:05:08.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6f4b00640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f00a0850 con 0x7fb6f00a1a10 2024-09-15T04:05:08.247 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6f4b00640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6f4b00640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f009c9a0 con 0x7fb6f009ee80 2024-09-15T04:05:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6f4b00640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6f4b00640 10 monclient(hunting): authenticate will time out at 1131.887423s 2024-09-15T04:05:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6eed9d640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb6f01369e0 0x7fb6f0138dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58970/0 (socket says 172.21.15.110:58970) 2024-09-15T04:05:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6eed9d640 1 -- 172.21.15.110:0/1633092986 learned_addr learned my addr 172.21.15.110:0/1633092986 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 1 --2- 172.21.15.110:0/1633092986 >> v2:172.21.15.177:3300/0 conn(0x7fb6f009ee80 0x7fb6f0131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 10 monclient(hunting): get_auth_request con 0x7fb6f009ee80 auth_method 0 2024-09-15T04:05:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 10 monclient(hunting): handle_auth_done global_id 5734 payload 274 2024-09-15T04:05:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 1 -- 172.21.15.110:0/1633092986 >> v1:172.21.15.177:6789/0 conn(0x7fb6f00a1a10 legacy=0x7fb6f009e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 1 -- 172.21.15.110:0/1633092986 >> v1:172.21.15.110:6789/0 conn(0x7fb6f01369e0 legacy=0x7fb6f0138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 10 monclient: _finish_hunting 0 2024-09-15T04:05:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 1 monclient: found mon.noname-c 2024-09-15T04:05:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.243+0000 7fb6edd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.177:3300/0 2024-09-15T04:05:08.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.244+0000 7fb6edd9b640 1 -- 172.21.15.110:0/1633092986 --> v2:172.21.15.177:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f009cc80 con 0x7fb6f009ee80 2024-09-15T04:05:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.246+0000 7fb6edd9b640 1 -- 172.21.15.110:0/1633092986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6f009ee80 msgr2=0x7fb6f0131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.246+0000 7fb6edd9b640 1 --2- 172.21.15.110:0/1633092986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6f009ee80 0x7fb6f0131b90 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4009850 tx=0x7fb6e40090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d436a46b4e0d642 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6ed59a640 1 -- 172.21.15.110:0/1633092986 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4032440 con 0x7fb6f009ee80 2024-09-15T04:05:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.256 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6ed59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6ed59a640 10 monclient: dump: 2024-09-15T04:05:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.257 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.257 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6ed59a640 10 monclient: _finish_auth 0 2024-09-15T04:05:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6ed59a640 1 -- 172.21.15.110:0/1633092986 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e40325a0 con 0x7fb6f009ee80 2024-09-15T04:05:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6f4b00640 5 monclient: authenticate success, global_id 5734 2024-09-15T04:05:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6f4b00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6ed59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6ed59a640 1 -- 172.21.15.110:0/1633092986 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4032940 con 0x7fb6f009ee80 2024-09-15T04:05:08.258 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6ed59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T04:05:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6ed59a640 10 monclient: dump: 2024-09-15T04:05:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.259 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.259 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6f4b00640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:08.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6f4b00640 4 set_mon_vals no callback set 2024-09-15T04:05:08.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6f4b00640 10 monclient: shutdown 2024-09-15T04:05:08.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6f4b00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:08.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6f4b00640 1 -- 172.21.15.110:0/1633092986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6f009ee80 msgr2=0x7fb6f0131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.247+0000 7fb6f4b00640 1 --2- 172.21.15.110:0/1633092986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6f009ee80 0x7fb6f0131b90 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4009850 tx=0x7fb6e40090f0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.248+0000 7fb6f4b00640 1 -- 172.21.15.110:0/1633092986 shutdown_connections 2024-09-15T04:05:08.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.248+0000 7fb6f4b00640 1 --2- 172.21.15.110:0/1633092986 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6f009ee80 0x7fb6f0131b90 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.248+0000 7fb6f4b00640 1 -- 172.21.15.110:0/1633092986 >> 172.21.15.110:0/1633092986 conn(0x7fb6f006e860 msgr2=0x7fb6f006ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:08.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.248+0000 7fb6f4b00640 1 -- 172.21.15.110:0/1633092986 shutdown_connections 2024-09-15T04:05:08.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.248+0000 7fb6f4b00640 1 -- 172.21.15.110:0/1633092986 wait complete. 2024-09-15T04:05:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.248+0000 7fb6f4b00640 10 monclient: build_initial_monmap 2024-09-15T04:05:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.248+0000 7fb6f4b00640 10 monclient: monmap: 2024-09-15T04:05:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:08.261 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:08.261 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.248+0000 7fb6f4b00640 1 Processor -- start 2024-09-15T04:05:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.248+0000 7fb6f4b00640 1 -- start start 2024-09-15T04:05:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.248+0000 7fb6f4b00640 10 monclient: init 2024-09-15T04:05:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T04:05:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6f009ee80 0x7fb6f0133890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 10 monclient(hunting): picked mon.noname-b con 0x7fb6f009ee80 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f00a1a10 0x7fb6f0133dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 10 monclient(hunting): picked mon.noname-a con 0x7fb6f00a1a10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6f01369e0 0x7fb6f0134310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 10 monclient(hunting): picked mon.noname-c con 0x7fb6f01369e0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:08.262 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f01307a0 con 0x7fb6f00a1a10 2024-09-15T04:05:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f0131200 con 0x7fb6f009ee80 2024-09-15T04:05:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f00a2e40 con 0x7fb6f01369e0 2024-09-15T04:05:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6f4b00640 10 monclient(hunting): authenticate will time out at 1131.893368s 2024-09-15T04:05:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6edd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f00a1a10 0x7fb6f0133dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6f01369e0 0x7fb6f0134310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6f01369e0 0x7fb6f0134310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35218/0 (socket says 172.21.15.110:35218) 2024-09-15T04:05:08.263 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6edd9b640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f00a1a10 0x7fb6f0133dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42870/0 (socket says 172.21.15.110:42870) 2024-09-15T04:05:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 1 -- 172.21.15.110:0/3375289462 learned_addr learned my addr 172.21.15.110:0/3375289462 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 10 monclient(hunting): get_auth_request con 0x7fb6f01369e0 auth_method 0 2024-09-15T04:05:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6edd9b640 10 monclient(hunting): get_auth_request con 0x7fb6f00a1a10 auth_method 0 2024-09-15T04:05:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.264 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 10 monclient(hunting): handle_auth_done global_id 74162 payload 824 2024-09-15T04:05:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6ee59c640 1 --2- 172.21.15.110:0/3375289462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6f009ee80 0x7fb6f0133890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 1 -- 172.21.15.110:0/3375289462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6f009ee80 msgr2=0x7fb6f0133890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 1 --2- 172.21.15.110:0/3375289462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6f009ee80 0x7fb6f0133890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 1 -- 172.21.15.110:0/3375289462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f00a1a10 msgr2=0x7fb6f0133dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 1 --2- 172.21.15.110:0/3375289462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f00a1a10 0x7fb6f0133dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 10 monclient: _finish_hunting 0 2024-09-15T04:05:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 1 monclient: found mon.noname-c 2024-09-15T04:05:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6eed9d640 1 -- 172.21.15.110:0/3375289462 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e4009390 con 0x7fb6f01369e0 2024-09-15T04:05:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6ee59c640 10 monclient: get_auth_request con 0x7fb6f009ee80 auth_method 0 2024-09-15T04:05:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6edd9b640 1 --2- 172.21.15.110:0/3375289462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f00a1a10 0x7fb6f0133dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.249+0000 7fb6ee59c640 1 --2- 172.21.15.110:0/3375289462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6f009ee80 0x7fb6f0133890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:05:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.251+0000 7fb6eed9d640 1 -- 172.21.15.110:0/3375289462 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6f01369e0 msgr2=0x7fb6f0134310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.251+0000 7fb6eed9d640 1 --2- 172.21.15.110:0/3375289462 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6f01369e0 0x7fb6f0134310 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb6e800c540 tx=0x7fb6e8004900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6de7fc640 1 -- 172.21.15.110:0/3375289462 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8005cb0 con 0x7fb6f01369e0 2024-09-15T04:05:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6de7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6de7fc640 10 monclient: dump: 2024-09-15T04:05:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.267 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.268 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6de7fc640 10 monclient: _finish_auth 0 2024-09-15T04:05:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6f4b00640 5 monclient: authenticate success, global_id 74162 2024-09-15T04:05:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6de7fc640 1 -- 172.21.15.110:0/3375289462 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e8005e10 con 0x7fb6f01369e0 2024-09-15T04:05:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6de7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6f4b00640 10 monclient: _renew_subs 2024-09-15T04:05:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6de7fc640 1 -- 172.21.15.110:0/3375289462 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8007320 con 0x7fb6f01369e0 2024-09-15T04:05:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6f4b00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6f4b00640 1 -- 172.21.15.110:0/3375289462 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f012d510 con 0x7fb6f01369e0 2024-09-15T04:05:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6de7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:08.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6de7fc640 10 monclient: dump: 2024-09-15T04:05:08.269 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.269 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.269 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.269 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.269 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.269 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.269 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.269 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.269 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.269 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6ecd99640 4 set_mon_vals no callback set 2024-09-15T04:05:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6f4b00640 10 monclient: _renew_subs 2024-09-15T04:05:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6f4b00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6f4b00640 1 -- 172.21.15.110:0/3375289462 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f0135d40 con 0x7fb6f01369e0 2024-09-15T04:05:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.252+0000 7fb6de7fc640 1 -- 172.21.15.110:0/3375289462 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fb6e801b3b0 con 0x7fb6f01369e0 2024-09-15T04:05:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.253+0000 7fb6f4b00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.253+0000 7fb6f4b00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.253+0000 7fb6f4b00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.253+0000 7fb6f4b00640 1 -- 172.21.15.110:0/3375289462 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f012d340 con 0x7fb6f01369e0 2024-09-15T04:05:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.253+0000 7fb6de7fc640 1 --2- 172.21.15.110:0/3375289462 >> v2:172.21.15.177:6812/28201 conn(0x7fb6d0038730 0x7fb6d003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.253+0000 7fb6de7fc640 1 -- 172.21.15.110:0/3375289462 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fb6e804aa60 con 0x7fb6f01369e0 2024-09-15T04:05:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.253+0000 7fb6ee59c640 1 --2- 172.21.15.110:0/3375289462 >> v2:172.21.15.177:6812/28201 conn(0x7fb6d0038730 0x7fb6d003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.253+0000 7fb6ee59c640 10 monclient: get_auth_request con 0x7fb6d0038730 auth_method 0 2024-09-15T04:05:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.256+0000 7fb6ee59c640 1 --2- 172.21.15.110:0/3375289462 >> v2:172.21.15.177:6812/28201 conn(0x7fb6d0038730 0x7fb6d003abe0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb6e000c330 tx=0x7fb6e00074d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.257+0000 7fb6de7fc640 1 -- 172.21.15.110:0/3375289462 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7fb6e8007750 con 0x7fb6f01369e0 2024-09-15T04:05:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.257+0000 7fb6de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:08.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.257+0000 7fb6de7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:08.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.412+0000 7fb6f4b00640 1 -- 172.21.15.110:0/3375289462 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6f00647e0 con 0x7fb6d0038730 2024-09-15T04:05:08.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.413+0000 7fb6de7fc640 1 -- 172.21.15.110:0/3375289462 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb6f00647e0 con 0x7fb6d0038730 2024-09-15T04:05:08.414 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 0 to deep-scrub 2024-09-15T04:05:08.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.415+0000 7fb6f4b00640 1 -- 172.21.15.110:0/3375289462 >> v2:172.21.15.177:6812/28201 conn(0x7fb6d0038730 msgr2=0x7fb6d003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.415+0000 7fb6f4b00640 1 --2- 172.21.15.110:0/3375289462 >> v2:172.21.15.177:6812/28201 conn(0x7fb6d0038730 0x7fb6d003abe0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb6e000c330 tx=0x7fb6e00074d0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.415+0000 7fb6f4b00640 10 monclient: shutdown 2024-09-15T04:05:08.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.415+0000 7fb6f4b00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:08.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.415+0000 7fb6f4b00640 1 -- 172.21.15.110:0/3375289462 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6f01369e0 msgr2=0x7fb6f0134310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.415+0000 7fb6f4b00640 1 --2- 172.21.15.110:0/3375289462 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6f01369e0 0x7fb6f0134310 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb6e800c540 tx=0x7fb6e8004900 comp rx=0 tx=0).stop 2024-09-15T04:05:08.416 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.415+0000 7fb6f4b00640 1 -- 172.21.15.110:0/3375289462 shutdown_connections 2024-09-15T04:05:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.415+0000 7fb6f4b00640 1 --2- 172.21.15.110:0/3375289462 >> v2:172.21.15.177:6812/28201 conn(0x7fb6d0038730 0x7fb6d003abe0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.415+0000 7fb6f4b00640 1 --2- 172.21.15.110:0/3375289462 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fb6f01369e0 0x7fb6f0134310 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.415+0000 7fb6f4b00640 1 --2- 172.21.15.110:0/3375289462 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6f00a1a10 0x7fb6f0133dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.415+0000 7fb6f4b00640 1 --2- 172.21.15.110:0/3375289462 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb6f009ee80 0x7fb6f0133890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.415+0000 7fb6f4b00640 1 -- 172.21.15.110:0/3375289462 >> 172.21.15.110:0/3375289462 conn(0x7fb6f006e860 msgr2=0x7fb6f012b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.415+0000 7fb6f4b00640 1 -- 172.21.15.110:0/3375289462 shutdown_connections 2024-09-15T04:05:08.417 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.415+0000 7fb6f4b00640 1 -- 172.21.15.110:0/3375289462 wait complete. 2024-09-15T04:05:08.426 INFO:tasks.ceph:Scrubbing osd.1 2024-09-15T04:05:08.426 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-15T04:05:08.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.522+0000 7f66b2d9d640 10 monclient: get_monmap_and_config 2024-09-15T04:05:08.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.522+0000 7f66b2d9d640 10 monclient: build_initial_monmap 2024-09-15T04:05:08.523 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.522+0000 7f66b2d9d640 10 monclient: monmap: 2024-09-15T04:05:08.523 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:08.523 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.524 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:08.523269+0000 2024-09-15T04:05:08.524 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:08.523269+0000 2024-09-15T04:05:08.524 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:08.524 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.524 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:08.524 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:08.524 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:08.524 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:08.524 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:08.524 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 1 Processor -- start 2024-09-15T04:05:08.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 1 -- start start 2024-09-15T04:05:08.524 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 10 monclient: init 2024-09-15T04:05:08.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:08.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-15T04:05:08.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66ac09e840 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:08.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f66ac131c30 0x7f66ac136a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f66ac131c30 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:08.526 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ac132050 0x7f66ac138fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f66ac132050 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:08.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ac09f320 con 0x7f66ac09e840 2024-09-15T04:05:08.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ac09ca50 con 0x7f66ac131c30 2024-09-15T04:05:08.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac09cd30 con 0x7f66ac132050 2024-09-15T04:05:08.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:08.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b2d9d640 10 monclient(hunting): authenticate will time out at 1132.167561s 2024-09-15T04:05:08.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ac132050 0x7f66ac138fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b0b39640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f66ac09e840 0x7f66ac09ec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58980/0 (socket says 172.21.15.110:58980) 2024-09-15T04:05:08.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b0b39640 1 -- 172.21.15.110:0/159119032 learned_addr learned my addr 172.21.15.110:0/159119032 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:08.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 10 monclient(hunting): get_auth_request con 0x7f66ac132050 auth_method 0 2024-09-15T04:05:08.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.527 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66abfff640 1 --2- 172.21.15.110:0/159119032 >> v2:172.21.15.177:3300/0 conn(0x7f66ac131c30 0x7f66ac136a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66abfff640 10 monclient(hunting): get_auth_request con 0x7f66ac131c30 auth_method 0 2024-09-15T04:05:08.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 10 monclient(hunting): handle_auth_done global_id 74165 payload 274 2024-09-15T04:05:08.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 1 -- 172.21.15.110:0/159119032 >> v2:172.21.15.177:3300/0 conn(0x7f66ac131c30 msgr2=0x7f66ac136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:08.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 1 --2- 172.21.15.110:0/159119032 >> v2:172.21.15.177:3300/0 conn(0x7f66ac131c30 0x7f66ac136a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 1 -- 172.21.15.110:0/159119032 >> v1:172.21.15.110:6789/0 conn(0x7f66ac09e840 legacy=0x7f66ac09ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:08.528 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 10 monclient: _finish_hunting 0 2024-09-15T04:05:08.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 1 monclient: found mon.noname-b 2024-09-15T04:05:08.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:08.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:05:08.529 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.523+0000 7f66b133a640 1 -- 172.21.15.110:0/159119032 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac09d010 con 0x7f66ac132050 2024-09-15T04:05:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66b133a640 1 -- 172.21.15.110:0/159119032 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ac132050 msgr2=0x7f66ac138fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66b133a640 1 --2- 172.21.15.110:0/159119032 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ac132050 0x7f66ac138fa0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009e90 tx=0x7f66a4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=861b1fe6252c3f96 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66ab7fe640 1 -- 172.21.15.110:0/159119032 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a4007760 con 0x7f66ac132050 2024-09-15T04:05:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:05:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66ab7fe640 10 monclient: dump: 2024-09-15T04:05:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.535 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.536 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.536 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.536 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66ab7fe640 10 monclient: _finish_auth 0 2024-09-15T04:05:08.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:08.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66ab7fe640 1 -- 172.21.15.110:0/159119032 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a402d050 con 0x7f66ac132050 2024-09-15T04:05:08.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66b2d9d640 5 monclient: authenticate success, global_id 74165 2024-09-15T04:05:08.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:08.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66b2d9d640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:08.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66ab7fe640 1 -- 172.21.15.110:0/159119032 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a4007af0 con 0x7f66ac132050 2024-09-15T04:05:08.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66b2d9d640 4 set_mon_vals no callback set 2024-09-15T04:05:08.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:08.536 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.526+0000 7f66ab7fe640 10 monclient: dump: 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.527+0000 7f66b2d9d640 10 monclient: shutdown 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.527+0000 7f66b2d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.527+0000 7f66b2d9d640 1 -- 172.21.15.110:0/159119032 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ac132050 msgr2=0x7f66ac138fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.527+0000 7f66b2d9d640 1 --2- 172.21.15.110:0/159119032 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ac132050 0x7f66ac138fa0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009e90 tx=0x7f66a4009780 comp rx=0 tx=0).stop 2024-09-15T04:05:08.537 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.527+0000 7f66b2d9d640 1 -- 172.21.15.110:0/159119032 shutdown_connections 2024-09-15T04:05:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.527+0000 7f66b2d9d640 1 --2- 172.21.15.110:0/159119032 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ac132050 0x7f66ac138fa0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.527+0000 7f66b2d9d640 1 --2- 172.21.15.110:0/159119032 >> v2:172.21.15.177:3300/0 conn(0x7f66ac131c30 0x7f66ac136a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.527+0000 7f66b2d9d640 1 -- 172.21.15.110:0/159119032 >> 172.21.15.110:0/159119032 conn(0x7f66ac063270 msgr2=0x7f66ac063670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.527+0000 7f66b2d9d640 1 -- 172.21.15.110:0/159119032 shutdown_connections 2024-09-15T04:05:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.527+0000 7f66b2d9d640 1 -- 172.21.15.110:0/159119032 wait complete. 2024-09-15T04:05:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.527+0000 7f66b2d9d640 10 monclient: build_initial_monmap 2024-09-15T04:05:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.527+0000 7f66b2d9d640 10 monclient: monmap: 2024-09-15T04:05:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.538 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 1 Processor -- start 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 1 -- start start 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 10 monclient: init 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ac09e840 0x7f66ac1341b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f66ac09e840 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66ac131c30 0x7f66ac1346f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f66ac131c30 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ac132050 0x7f66ac134c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f66ac132050 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:08.539 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac131270 con 0x7f66ac132050 2024-09-15T04:05:08.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac138910 con 0x7f66ac131c30 2024-09-15T04:05:08.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac137bb0 con 0x7f66ac09e840 2024-09-15T04:05:08.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:08.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b2d9d640 10 monclient(hunting): authenticate will time out at 1132.172848s 2024-09-15T04:05:08.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b0b39640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ac09e840 0x7f66ac1341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b133a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ac132050 0x7f66ac134c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b0b39640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ac09e840 0x7f66ac1341b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35232/0 (socket says 172.21.15.110:35232) 2024-09-15T04:05:08.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b0b39640 1 -- 172.21.15.110:0/1654993015 learned_addr learned my addr 172.21.15.110:0/1654993015 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:08.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b0b39640 10 monclient(hunting): get_auth_request con 0x7f66ac09e840 auth_method 0 2024-09-15T04:05:08.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b0b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.540 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b0b39640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b0b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b133a640 10 monclient(hunting): get_auth_request con 0x7f66ac132050 auth_method 0 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b133a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b133a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.528+0000 7f66b133a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b0b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b0b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b0b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b133a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b133a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b133a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66abfff640 1 --2- 172.21.15.110:0/1654993015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66ac131c30 0x7f66ac1346f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66abfff640 10 monclient(hunting): get_auth_request con 0x7f66ac131c30 auth_method 0 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.541 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66abfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b0b39640 10 monclient(hunting): handle_auth_done global_id 74168 payload 824 2024-09-15T04:05:08.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b0b39640 1 -- 172.21.15.110:0/1654993015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66ac131c30 msgr2=0x7f66ac1346f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b0b39640 1 --2- 172.21.15.110:0/1654993015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66ac131c30 0x7f66ac1346f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b0b39640 1 -- 172.21.15.110:0/1654993015 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ac132050 msgr2=0x7f66ac134c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b0b39640 1 --2- 172.21.15.110:0/1654993015 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ac132050 0x7f66ac134c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b0b39640 10 monclient: _finish_hunting 0 2024-09-15T04:05:08.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b0b39640 1 monclient: found mon.noname-c 2024-09-15T04:05:08.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b0b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:08.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b0b39640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:08.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b0b39640 1 -- 172.21.15.110:0/1654993015 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a4009a20 con 0x7f66ac09e840 2024-09-15T04:05:08.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66abfff640 1 --2- 172.21.15.110:0/1654993015 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f66ac131c30 0x7f66ac1346f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T04:05:08.542 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.529+0000 7f66b133a640 1 --2- 172.21.15.110:0/1654993015 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f66ac132050 0x7f66ac134c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66b0b39640 1 -- 172.21.15.110:0/1654993015 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ac09e840 msgr2=0x7f66ac1341b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66b0b39640 1 --2- 172.21.15.110:0/1654993015 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ac09e840 0x7f66ac1341b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f66a000c610 tx=0x7f66a0004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66a8ff9640 1 -- 172.21.15.110:0/1654993015 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a0005cb0 con 0x7f66ac09e840 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66a8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66a8ff9640 10 monclient: dump: 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.543 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.544 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66a8ff9640 10 monclient: _finish_auth 0 2024-09-15T04:05:08.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:08.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66b2d9d640 5 monclient: authenticate success, global_id 74168 2024-09-15T04:05:08.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66a8ff9640 1 -- 172.21.15.110:0/1654993015 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a0005e30 con 0x7f66ac09e840 2024-09-15T04:05:08.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66b2d9d640 10 monclient: _renew_subs 2024-09-15T04:05:08.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66b2d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:08.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66b2d9d640 1 -- 172.21.15.110:0/1654993015 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ac1375f0 con 0x7f66ac09e840 2024-09-15T04:05:08.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:08.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66a8ff9640 1 -- 172.21.15.110:0/1654993015 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a0007320 con 0x7f66ac09e840 2024-09-15T04:05:08.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:08.544 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66a8ff9640 10 monclient: dump: 2024-09-15T04:05:08.544 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.545 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.545 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.545 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.545 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.545 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.545 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.545 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.545 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.545 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66aaffd640 4 set_mon_vals no callback set 2024-09-15T04:05:08.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66b2d9d640 10 monclient: _renew_subs 2024-09-15T04:05:08.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66b2d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:08.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.531+0000 7f66b2d9d640 1 -- 172.21.15.110:0/1654993015 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ac136690 con 0x7f66ac09e840 2024-09-15T04:05:08.545 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.532+0000 7f66b2d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:08.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.532+0000 7f66b2d9d640 1 -- 172.21.15.110:0/1654993015 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f66ac136910 con 0x7f66ac09e840 2024-09-15T04:05:08.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.532+0000 7f66a8ff9640 1 -- 172.21.15.110:0/1654993015 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f66a001b3b0 con 0x7f66ac09e840 2024-09-15T04:05:08.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.532+0000 7f66a8ff9640 1 --2- 172.21.15.110:0/1654993015 >> v2:172.21.15.177:6812/28201 conn(0x7f6690038700 0x7f669003abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.532+0000 7f66a8ff9640 1 -- 172.21.15.110:0/1654993015 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f66a004aad0 con 0x7f66ac09e840 2024-09-15T04:05:08.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.532+0000 7f66a8ff9640 1 --2- 172.21.15.110:0/1654993015 >> v2:172.21.15.110:6800/28369 conn(0x7f669003e220 0x7f6690040630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.532+0000 7f66a8ff9640 1 -- 172.21.15.110:0/1654993015 --> v2:172.21.15.110:6800/28369 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6690040ca0 con 0x7f669003e220 2024-09-15T04:05:08.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.532+0000 7f66a8ff9640 1 -- 172.21.15.110:0/1654993015 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=315) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66a004f2a0 con 0x7f66ac09e840 2024-09-15T04:05:08.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.532+0000 7f66a8ff9640 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-15T04:05:08.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.532+0000 7f66b133a640 1 --2- 172.21.15.110:0/1654993015 >> v2:172.21.15.110:6800/28369 conn(0x7f669003e220 0x7f6690040630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.532+0000 7f66b133a640 10 monclient: get_auth_request con 0x7f669003e220 auth_method 0 2024-09-15T04:05:08.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.532+0000 7f66abfff640 1 --2- 172.21.15.110:0/1654993015 >> v2:172.21.15.177:6812/28201 conn(0x7f6690038700 0x7f669003abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.546 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.532+0000 7f66abfff640 10 monclient: get_auth_request con 0x7f6690038700 auth_method 0 2024-09-15T04:05:08.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.535+0000 7f66abfff640 1 --2- 172.21.15.110:0/1654993015 >> v2:172.21.15.177:6812/28201 conn(0x7f6690038700 0x7f669003abb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f669800bb80 tx=0x7f6698008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.537+0000 7f66b133a640 1 --2- 172.21.15.110:0/1654993015 >> v2:172.21.15.110:6800/28369 conn(0x7f669003e220 0x7f6690040630 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.547 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.537+0000 7f66a8ff9640 1 -- 172.21.15.110:0/1654993015 <== osd.1 v2:172.21.15.110:6800/28369 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f6690040ca0 con 0x7f669003e220 2024-09-15T04:05:08.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66b2d9d640 1 -- 172.21.15.110:0/1654993015 --> v2:172.21.15.110:6800/28369 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f66ac070950 con 0x7f669003e220 2024-09-15T04:05:08.548 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66a8ff9640 1 -- 172.21.15.110:0/1654993015 <== osd.1 v2:172.21.15.110:6800/28369 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f66ac070950 con 0x7f669003e220 2024-09-15T04:05:08.549 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:05:08.549 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T04:05:08.549 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:05:08.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66b2d9d640 1 -- 172.21.15.110:0/1654993015 >> v2:172.21.15.110:6800/28369 conn(0x7f669003e220 msgr2=0x7f6690040630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66b2d9d640 1 --2- 172.21.15.110:0/1654993015 >> v2:172.21.15.110:6800/28369 conn(0x7f669003e220 0x7f6690040630 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66b2d9d640 1 -- 172.21.15.110:0/1654993015 >> v2:172.21.15.177:6812/28201 conn(0x7f6690038700 msgr2=0x7f669003abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66b2d9d640 1 --2- 172.21.15.110:0/1654993015 >> v2:172.21.15.177:6812/28201 conn(0x7f6690038700 0x7f669003abb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f669800bb80 tx=0x7f6698008040 comp rx=0 tx=0).stop 2024-09-15T04:05:08.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66b2d9d640 10 monclient: shutdown 2024-09-15T04:05:08.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66b2d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:08.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66b2d9d640 1 -- 172.21.15.110:0/1654993015 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ac09e840 msgr2=0x7f66ac1341b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66b2d9d640 1 --2- 172.21.15.110:0/1654993015 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f66ac09e840 0x7f66ac1341b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f66a000c610 tx=0x7f66a0004960 comp rx=0 tx=0).stop 2024-09-15T04:05:08.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66b133a640 1 -- 172.21.15.110:0/1654993015 reap_dead start 2024-09-15T04:05:08.550 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66b2d9d640 1 -- 172.21.15.110:0/1654993015 shutdown_connections 2024-09-15T04:05:08.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66b2d9d640 1 -- 172.21.15.110:0/1654993015 >> 172.21.15.110:0/1654993015 conn(0x7f66ac063270 msgr2=0x7f66ac0a5780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:08.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66b2d9d640 1 -- 172.21.15.110:0/1654993015 shutdown_connections 2024-09-15T04:05:08.551 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.548+0000 7f66b2d9d640 1 -- 172.21.15.110:0/1654993015 wait complete. 2024-09-15T04:05:08.559 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-15T04:05:08.652 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.651+0000 7f4a8e59d640 10 monclient: get_monmap_and_config 2024-09-15T04:05:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.651+0000 7f4a8e59d640 10 monclient: build_initial_monmap 2024-09-15T04:05:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.651+0000 7f4a8e59d640 10 monclient: monmap: 2024-09-15T04:05:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.653 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:08.652863+0000 2024-09-15T04:05:08.654 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:08.652863+0000 2024-09-15T04:05:08.654 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:08.654 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.654 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:08.654 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:08.654 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:08.655 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:08.655 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:08.655 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.652+0000 7f4a8e59d640 1 Processor -- start 2024-09-15T04:05:08.667 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.652+0000 7f4a8e59d640 1 -- start start 2024-09-15T04:05:08.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.652+0000 7f4a8e59d640 10 monclient: init 2024-09-15T04:05:08.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a8e59d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:08.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a8e59d640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-15T04:05:08.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a8e59d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f4a880a3480 0x7f4a8806e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a8e59d640 10 monclient(hunting): picked mon.noname-c con 0x7f4a880a3480 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:08.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a8e59d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4a8812f5e0 0x7f4a8812fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a8e59d640 10 monclient(hunting): picked mon.noname-a con 0x7f4a8812f5e0 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:08.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a8e59d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a8812ff60 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:08.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a8e59d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8809f290 con 0x7f4a8812ff60 2024-09-15T04:05:08.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a8e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a8e59d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a880a3850 con 0x7f4a8812f5e0 2024-09-15T04:05:08.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a8e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.668 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a8e59d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a881303e0 con 0x7f4a880a3480 2024-09-15T04:05:08.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a8e59d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:08.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a8e59d640 10 monclient(hunting): authenticate will time out at 1132.297221s 2024-09-15T04:05:08.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4a8812f5e0 0x7f4a8812fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4a8812f5e0 0x7f4a8812fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42886/0 (socket says 172.21.15.110:42886) 2024-09-15T04:05:08.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 1 -- 172.21.15.110:0/2150826161 learned_addr learned my addr 172.21.15.110:0/2150826161 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:08.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 10 monclient(hunting): get_auth_request con 0x7f4a8812f5e0 auth_method 0 2024-09-15T04:05:08.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a87fff640 1 --2- 172.21.15.110:0/2150826161 >> v2:172.21.15.177:3300/0 conn(0x7f4a880a3480 0x7f4a8806e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.669 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a87fff640 10 monclient(hunting): get_auth_request con 0x7f4a880a3480 auth_method 0 2024-09-15T04:05:08.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 10 monclient(hunting): handle_auth_done global_id 55224 payload 274 2024-09-15T04:05:08.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 1 -- 172.21.15.110:0/2150826161 >> v2:172.21.15.177:3300/0 conn(0x7f4a880a3480 msgr2=0x7f4a8806e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:08.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 1 --2- 172.21.15.110:0/2150826161 >> v2:172.21.15.177:3300/0 conn(0x7f4a880a3480 0x7f4a8806e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 1 -- 172.21.15.110:0/2150826161 >> v1:172.21.15.177:6789/0 conn(0x7f4a8812ff60 legacy=0x7f4a8809d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:08.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 10 monclient: _finish_hunting 0 2024-09-15T04:05:08.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 1 monclient: found mon.noname-a 2024-09-15T04:05:08.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:08.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:08.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.653+0000 7f4a877fe640 1 -- 172.21.15.110:0/2150826161 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8809f050 con 0x7f4a8812f5e0 2024-09-15T04:05:08.670 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a877fe640 1 -- 172.21.15.110:0/2150826161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8812f5e0 msgr2=0x7f4a8812fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:08.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a877fe640 1 --2- 172.21.15.110:0/2150826161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8812f5e0 0x7f4a8812fa20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4a8806f1b0 tx=0x7f4a7c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4544fcd05350d1c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a86ffd640 1 -- 172.21.15.110:0/2150826161 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c032440 con 0x7f4a8812f5e0 2024-09-15T04:05:08.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a86ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:08.671 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a86ffd640 10 monclient: dump: 2024-09-15T04:05:08.671 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.671 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.671 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.671 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.671 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.671 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.671 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.671 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.671 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.672 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a86ffd640 10 monclient: _finish_auth 0 2024-09-15T04:05:08.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:08.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a86ffd640 1 -- 172.21.15.110:0/2150826161 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7c0325a0 con 0x7f4a8812f5e0 2024-09-15T04:05:08.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a86ffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:08.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a86ffd640 1 -- 172.21.15.110:0/2150826161 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c032940 con 0x7f4a8812f5e0 2024-09-15T04:05:08.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a8e59d640 5 monclient: authenticate success, global_id 55224 2024-09-15T04:05:08.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a8e59d640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:08.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a8e59d640 4 set_mon_vals no callback set 2024-09-15T04:05:08.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:08.672 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.656+0000 7f4a86ffd640 10 monclient: dump: 2024-09-15T04:05:08.672 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.672 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.673 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.673 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.673 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.673 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.673 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.673 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.673 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.673 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.657+0000 7f4a8e59d640 10 monclient: shutdown 2024-09-15T04:05:08.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.657+0000 7f4a8e59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:08.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.657+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2150826161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8812f5e0 msgr2=0x7f4a8812fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.657+0000 7f4a8e59d640 1 --2- 172.21.15.110:0/2150826161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8812f5e0 0x7f4a8812fa20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4a8806f1b0 tx=0x7f4a7c0091d0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.657+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2150826161 shutdown_connections 2024-09-15T04:05:08.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.657+0000 7f4a8e59d640 1 --2- 172.21.15.110:0/2150826161 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8812f5e0 0x7f4a8812fa20 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.657+0000 7f4a8e59d640 1 --2- 172.21.15.110:0/2150826161 >> v2:172.21.15.177:3300/0 conn(0x7f4a880a3480 0x7f4a8806e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.657+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2150826161 >> 172.21.15.110:0/2150826161 conn(0x7f4a880635b0 msgr2=0x7f4a880a2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:08.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.657+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2150826161 shutdown_connections 2024-09-15T04:05:08.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.657+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2150826161 wait complete. 2024-09-15T04:05:08.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.657+0000 7f4a8e59d640 10 monclient: build_initial_monmap 2024-09-15T04:05:08.674 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.657+0000 7f4a8e59d640 10 monclient: monmap: 2024-09-15T04:05:08.674 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:08.674 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:08.674 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:08.674 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:08.674 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:08.674 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.674 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:08.674 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.657+0000 7f4a8e59d640 1 Processor -- start 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 1 -- start start 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 10 monclient: init 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a880a3480 0x7f4a8812c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 10 monclient(hunting): picked mon.noname-c con 0x7f4a880a3480 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a8812f5e0 0x7f4a8813e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 10 monclient(hunting): picked mon.noname-b con 0x7f4a8812f5e0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8812ff60 0x7f4a8809ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 10 monclient(hunting): picked mon.noname-a con 0x7f4a8812ff60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.675 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8809c200 con 0x7f4a8812ff60 2024-09-15T04:05:08.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8806c8e0 con 0x7f4a8812f5e0 2024-09-15T04:05:08.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a88119390 con 0x7f4a880a3480 2024-09-15T04:05:08.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:08.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8e59d640 10 monclient(hunting): authenticate will time out at 1132.302537s 2024-09-15T04:05:08.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a87fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a880a3480 0x7f4a8812c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8812ff60 0x7f4a8809ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a87fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a880a3480 0x7f4a8812c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35242/0 (socket says 172.21.15.110:35242) 2024-09-15T04:05:08.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 10 monclient(hunting): get_auth_request con 0x7f4a8812ff60 auth_method 0 2024-09-15T04:05:08.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a87fff640 1 -- 172.21.15.110:0/2268285766 learned_addr learned my addr 172.21.15.110:0/2268285766 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:08.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a87fff640 10 monclient(hunting): get_auth_request con 0x7f4a880a3480 auth_method 0 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a87fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 10 monclient(hunting): handle_auth_done global_id 55227 payload 824 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a877fe640 1 --2- 172.21.15.110:0/2268285766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a8812f5e0 0x7f4a8813e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 1 -- 172.21.15.110:0/2268285766 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a880a3480 msgr2=0x7f4a8812c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 1 --2- 172.21.15.110:0/2268285766 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a880a3480 0x7f4a8812c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 1 -- 172.21.15.110:0/2268285766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a8812f5e0 msgr2=0x7f4a8813e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 1 --2- 172.21.15.110:0/2268285766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a8812f5e0 0x7f4a8813e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 10 monclient: _finish_hunting 0 2024-09-15T04:05:08.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 1 monclient: found mon.noname-a 2024-09-15T04:05:08.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:08.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:08.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a8cb3a640 1 -- 172.21.15.110:0/2268285766 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c009450 con 0x7f4a8812ff60 2024-09-15T04:05:08.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.658+0000 7f4a87fff640 1 --2- 172.21.15.110:0/2268285766 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a880a3480 0x7f4a8812c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:08.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a8cb3a640 1 -- 172.21.15.110:0/2268285766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8812ff60 msgr2=0x7f4a8809ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:08.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a8cb3a640 1 --2- 172.21.15.110:0/2268285766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8812ff60 0x7f4a8809ad90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000c7e0 tx=0x7f4a800025b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a6ffff640 1 -- 172.21.15.110:0/2268285766 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8000a850 con 0x7f4a8812ff60 2024-09-15T04:05:08.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a6ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:08.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a6ffff640 10 monclient: dump: 2024-09-15T04:05:08.679 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.679 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.679 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.679 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.679 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.679 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.679 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.679 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.679 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.679 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a6ffff640 10 monclient: _finish_auth 0 2024-09-15T04:05:08.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:08.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a6ffff640 1 -- 172.21.15.110:0/2268285766 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8000a9d0 con 0x7f4a8812ff60 2024-09-15T04:05:08.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a8e59d640 5 monclient: authenticate success, global_id 55227 2024-09-15T04:05:08.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a6ffff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:08.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a8e59d640 10 monclient: _renew_subs 2024-09-15T04:05:08.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a6ffff640 1 -- 172.21.15.110:0/2268285766 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8000ad90 con 0x7f4a8812ff60 2024-09-15T04:05:08.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a8e59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:08.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2268285766 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a88119a20 con 0x7f4a8812ff60 2024-09-15T04:05:08.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:08.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a6ffff640 10 monclient: dump: 2024-09-15T04:05:08.680 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.680 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.680 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.680 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.680 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.680 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a867fc640 4 set_mon_vals no callback set 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a8e59d640 10 monclient: _renew_subs 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a8e59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.661+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2268285766 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8811ac60 con 0x7f4a8812ff60 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.662+0000 7f4a6ffff640 1 -- 172.21.15.110:0/2268285766 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f4a80012630 con 0x7f4a8812ff60 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.662+0000 7f4a8e59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.662+0000 7f4a8e59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.662+0000 7f4a8e59d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.662+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2268285766 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a880647e0 con 0x7f4a8812ff60 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.662+0000 7f4a6ffff640 1 --2- 172.21.15.110:0/2268285766 >> v2:172.21.15.177:6812/28201 conn(0x7f4a74038700 0x7f4a7403abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.681 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.662+0000 7f4a6ffff640 1 -- 172.21.15.110:0/2268285766 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4a8004b810 con 0x7f4a8812ff60 2024-09-15T04:05:08.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.662+0000 7f4a87fff640 1 --2- 172.21.15.110:0/2268285766 >> v2:172.21.15.177:6812/28201 conn(0x7f4a74038700 0x7f4a7403abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.662+0000 7f4a87fff640 10 monclient: get_auth_request con 0x7f4a74038700 auth_method 0 2024-09-15T04:05:08.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.665+0000 7f4a87fff640 1 --2- 172.21.15.110:0/2268285766 >> v2:172.21.15.177:6812/28201 conn(0x7f4a74038700 0x7f4a7403abb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4a78010370 tx=0x7f4a78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.666+0000 7f4a6ffff640 1 -- 172.21.15.110:0/2268285766 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f4a80022d70 con 0x7f4a8812ff60 2024-09-15T04:05:08.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.666+0000 7f4a6ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:08.682 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.666+0000 7f4a6ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:08.823 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.823+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2268285766 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a8806a450 con 0x7f4a74038700 2024-09-15T04:05:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.823+0000 7f4a6ffff640 1 -- 172.21.15.110:0/2268285766 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4a8806a450 con 0x7f4a74038700 2024-09-15T04:05:08.824 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 1 to deep-scrub 2024-09-15T04:05:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.825+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2268285766 >> v2:172.21.15.177:6812/28201 conn(0x7f4a74038700 msgr2=0x7f4a7403abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.825+0000 7f4a8e59d640 1 --2- 172.21.15.110:0/2268285766 >> v2:172.21.15.177:6812/28201 conn(0x7f4a74038700 0x7f4a7403abb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4a78010370 tx=0x7f4a78008040 comp rx=0 tx=0).stop 2024-09-15T04:05:08.826 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.825+0000 7f4a8e59d640 10 monclient: shutdown 2024-09-15T04:05:08.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.825+0000 7f4a8e59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:08.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.825+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2268285766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8812ff60 msgr2=0x7f4a8809ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.825+0000 7f4a8e59d640 1 --2- 172.21.15.110:0/2268285766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8812ff60 0x7f4a8809ad90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4a8000c7e0 tx=0x7f4a800025b0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.825+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2268285766 shutdown_connections 2024-09-15T04:05:08.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.825+0000 7f4a8e59d640 1 --2- 172.21.15.110:0/2268285766 >> v2:172.21.15.177:6812/28201 conn(0x7f4a74038700 0x7f4a7403abb0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.825+0000 7f4a8e59d640 1 --2- 172.21.15.110:0/2268285766 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4a8812ff60 0x7f4a8809ad90 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.825+0000 7f4a8e59d640 1 --2- 172.21.15.110:0/2268285766 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4a8812f5e0 0x7f4a8813e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.825+0000 7f4a8e59d640 1 --2- 172.21.15.110:0/2268285766 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f4a880a3480 0x7f4a8812c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.825+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2268285766 >> 172.21.15.110:0/2268285766 conn(0x7f4a880635b0 msgr2=0x7f4a880a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:08.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.825+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2268285766 shutdown_connections 2024-09-15T04:05:08.827 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.825+0000 7f4a8e59d640 1 -- 172.21.15.110:0/2268285766 wait complete. 2024-09-15T04:05:08.836 INFO:tasks.ceph:Scrubbing osd.2 2024-09-15T04:05:08.837 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-15T04:05:08.933 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.932+0000 7f016184f640 10 monclient: get_monmap_and_config 2024-09-15T04:05:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.933+0000 7f016184f640 10 monclient: build_initial_monmap 2024-09-15T04:05:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.933+0000 7f016184f640 10 monclient: monmap: 2024-09-15T04:05:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:08.934003+0000 2024-09-15T04:05:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:08.934003+0000 2024-09-15T04:05:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:08.934 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:08.934 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.936 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.933+0000 7f016184f640 1 Processor -- start 2024-09-15T04:05:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.933+0000 7f016184f640 1 -- start start 2024-09-15T04:05:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.933+0000 7f016184f640 10 monclient: init 2024-09-15T04:05:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f016184f640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f016184f640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-15T04:05:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f016184f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f015c131630 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f016184f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f015c06e5a0 0x7f015c06e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f016184f640 10 monclient(hunting): picked mon.noname-b con 0x7f015c06e5a0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f016184f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f015c06eeb0 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f016184f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015c068390 con 0x7f015c06eeb0 2024-09-15T04:05:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f016184f640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015c069570 con 0x7f015c131630 2024-09-15T04:05:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f016184f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f016184f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c09f3b0 con 0x7f015c06e5a0 2024-09-15T04:05:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f016184f640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:08.937 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f016184f640 10 monclient(hunting): authenticate will time out at 1132.578238s 2024-09-15T04:05:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f015c06e5a0 0x7f015c06e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f015b7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f015c06eeb0 0x7f015c138970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:58992/0 (socket says 172.21.15.110:58992) 2024-09-15T04:05:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f015b7fe640 1 -- 172.21.15.110:0/888438786 learned_addr learned my addr 172.21.15.110:0/888438786 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 10 monclient(hunting): get_auth_request con 0x7f015c06e5a0 auth_method 0 2024-09-15T04:05:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 10 monclient(hunting): handle_auth_done global_id 74174 payload 274 2024-09-15T04:05:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 1 -- 172.21.15.110:0/888438786 >> v1:172.21.15.177:6789/0 conn(0x7f015c131630 legacy=0x7f015c131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 1 -- 172.21.15.110:0/888438786 >> v1:172.21.15.110:6789/0 conn(0x7f015c06eeb0 legacy=0x7f015c138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:08.938 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 10 monclient: _finish_hunting 0 2024-09-15T04:05:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 1 monclient: found mon.noname-b 2024-09-15T04:05:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:05:08.939 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.934+0000 7f01527fc640 1 -- 172.21.15.110:0/888438786 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c069850 con 0x7f015c06e5a0 2024-09-15T04:05:08.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f01527fc640 1 -- 172.21.15.110:0/888438786 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f015c06e5a0 msgr2=0x7f015c06e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:08.945 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f01527fc640 1 --2- 172.21.15.110:0/888438786 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f015c06e5a0 0x7f015c06e970 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0144009880 tx=0x7f01440090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5169803b051e9c29 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f015a7fc640 1 -- 172.21.15.110:0/888438786 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0144032440 con 0x7f015c06e5a0 2024-09-15T04:05:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f015a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f015a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:05:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f015a7fc640 10 monclient: dump: 2024-09-15T04:05:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.946 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.947 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f015a7fc640 10 monclient: _finish_auth 0 2024-09-15T04:05:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f015a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f016184f640 5 monclient: authenticate success, global_id 74174 2024-09-15T04:05:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f016184f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f015a7fc640 1 -- 172.21.15.110:0/888438786 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01440325a0 con 0x7f015c06e5a0 2024-09-15T04:05:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f015a7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f016184f640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f016184f640 4 set_mon_vals no callback set 2024-09-15T04:05:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f015a7fc640 1 -- 172.21.15.110:0/888438786 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0144032940 con 0x7f015c06e5a0 2024-09-15T04:05:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f015a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f015a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:08.947 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.937+0000 7f015a7fc640 10 monclient: dump: 2024-09-15T04:05:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.948 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.938+0000 7f016184f640 10 monclient: shutdown 2024-09-15T04:05:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.938+0000 7f016184f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.938+0000 7f016184f640 1 -- 172.21.15.110:0/888438786 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f015c06e5a0 msgr2=0x7f015c06e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.948 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.938+0000 7f016184f640 1 --2- 172.21.15.110:0/888438786 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f015c06e5a0 0x7f015c06e970 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0144009880 tx=0x7f01440090f0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.938+0000 7f016184f640 1 -- 172.21.15.110:0/888438786 shutdown_connections 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.938+0000 7f016184f640 1 --2- 172.21.15.110:0/888438786 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f015c06e5a0 0x7f015c06e970 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.938+0000 7f016184f640 1 -- 172.21.15.110:0/888438786 >> 172.21.15.110:0/888438786 conn(0x7f015c0639f0 msgr2=0x7f015c0a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.938+0000 7f016184f640 1 -- 172.21.15.110:0/888438786 shutdown_connections 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.938+0000 7f016184f640 1 -- 172.21.15.110:0/888438786 wait complete. 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.938+0000 7f016184f640 10 monclient: build_initial_monmap 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.938+0000 7f016184f640 10 monclient: monmap: 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:08.949 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 1 Processor -- start 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 1 -- start start 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 10 monclient: init 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f015c06e5a0 0x7f015c135e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 10 monclient(hunting): picked mon.noname-a con 0x7f015c06e5a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f015c06eeb0 0x7f015c1363b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 10 monclient(hunting): picked mon.noname-b con 0x7f015c06eeb0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f015c131630 0x7f015c1368f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 10 monclient(hunting): picked mon.noname-c con 0x7f015c131630 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.950 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c1384f0 con 0x7f015c06e5a0 2024-09-15T04:05:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c0688c0 con 0x7f015c06eeb0 2024-09-15T04:05:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c137e30 con 0x7f015c131630 2024-09-15T04:05:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f016184f640 10 monclient(hunting): authenticate will time out at 1132.583755s 2024-09-15T04:05:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f015c06e5a0 0x7f015c135e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015affd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f015c06e5a0 0x7f015c135e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:42914/0 (socket says 172.21.15.110:42914) 2024-09-15T04:05:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015affd640 1 -- 172.21.15.110:0/1791936888 learned_addr learned my addr 172.21.15.110:0/1791936888 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015affd640 10 monclient(hunting): get_auth_request con 0x7f015c06e5a0 auth_method 0 2024-09-15T04:05:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.951 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015b7fe640 1 --2- 172.21.15.110:0/1791936888 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f015c131630 0x7f015c1368f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015b7fe640 10 monclient(hunting): get_auth_request con 0x7f015c131630 auth_method 0 2024-09-15T04:05:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f01527fc640 1 --2- 172.21.15.110:0/1791936888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f015c06eeb0 0x7f015c1363b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f01527fc640 10 monclient(hunting): get_auth_request con 0x7f015c06eeb0 auth_method 0 2024-09-15T04:05:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f01527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f01527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f01527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:08.952 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.939+0000 7f015b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.940+0000 7f01527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.940+0000 7f01527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.940+0000 7f01527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.940+0000 7f015affd640 10 monclient(hunting): handle_auth_done global_id 55230 payload 824 2024-09-15T04:05:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.940+0000 7f015affd640 1 -- 172.21.15.110:0/1791936888 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f015c131630 msgr2=0x7f015c1368f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.940+0000 7f015affd640 1 --2- 172.21.15.110:0/1791936888 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f015c131630 0x7f015c1368f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.940+0000 7f015affd640 1 -- 172.21.15.110:0/1791936888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f015c06eeb0 msgr2=0x7f015c1363b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.940+0000 7f015affd640 1 --2- 172.21.15.110:0/1791936888 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f015c06eeb0 0x7f015c1363b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.940+0000 7f015affd640 10 monclient: _finish_hunting 0 2024-09-15T04:05:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.940+0000 7f015affd640 1 monclient: found mon.noname-a 2024-09-15T04:05:08.953 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.940+0000 7f015affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.940+0000 7f015affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.940+0000 7f015affd640 1 -- 172.21.15.110:0/1791936888 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0144009390 con 0x7f015c06e5a0 2024-09-15T04:05:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.940+0000 7f015b7fe640 1 --2- 172.21.15.110:0/1791936888 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f015c131630 0x7f015c1368f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:08.954 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f015affd640 1 -- 172.21.15.110:0/1791936888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f015c06e5a0 msgr2=0x7f015c135e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f015affd640 1 --2- 172.21.15.110:0/1791936888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f015c06e5a0 0x7f015c135e70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f014c00c5e0 tx=0x7f014c007ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f01537fe640 1 -- 172.21.15.110:0/1791936888 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c012840 con 0x7f015c06e5a0 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f01537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f01537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f01537fe640 10 monclient: dump: 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.955 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f01537fe640 10 monclient: _finish_auth 0 2024-09-15T04:05:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f01537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f016184f640 5 monclient: authenticate success, global_id 55230 2024-09-15T04:05:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f01537fe640 1 -- 172.21.15.110:0/1791936888 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c0129a0 con 0x7f015c06e5a0 2024-09-15T04:05:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f01537fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f01537fe640 1 -- 172.21.15.110:0/1791936888 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c012d60 con 0x7f015c06e5a0 2024-09-15T04:05:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f016184f640 10 monclient: _renew_subs 2024-09-15T04:05:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f016184f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f016184f640 1 -- 172.21.15.110:0/1791936888 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c139450 con 0x7f015c06e5a0 2024-09-15T04:05:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f0159ffb640 4 set_mon_vals no callback set 2024-09-15T04:05:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f01537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f01537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f01537fe640 10 monclient: dump: 2024-09-15T04:05:08.956 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:08.957 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:08.957 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.957 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:08.957 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:08.957 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:08.957 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:08.957 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:08.957 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:08.957 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:08.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f016184f640 10 monclient: _renew_subs 2024-09-15T04:05:08.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f016184f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:08.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.942+0000 7f016184f640 1 -- 172.21.15.110:0/1791936888 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c1311d0 con 0x7f015c06e5a0 2024-09-15T04:05:08.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.943+0000 7f016184f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:08.957 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.943+0000 7f016184f640 1 -- 172.21.15.110:0/1791936888 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f015c070950 con 0x7f015c06e5a0 2024-09-15T04:05:08.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.943+0000 7f01537fe640 1 -- 172.21.15.110:0/1791936888 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f014c011590 con 0x7f015c06e5a0 2024-09-15T04:05:08.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.943+0000 7f01537fe640 1 --2- 172.21.15.110:0/1791936888 >> v2:172.21.15.177:6812/28201 conn(0x7f013c038700 0x7f013c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.943+0000 7f01537fe640 1 -- 172.21.15.110:0/1791936888 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f014c04b690 con 0x7f015c06e5a0 2024-09-15T04:05:08.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.943+0000 7f01537fe640 1 --2- 172.21.15.110:0/1791936888 >> v2:172.21.15.110:6804/28368 conn(0x7f013c03e220 0x7f013c040630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:08.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.943+0000 7f01537fe640 1 -- 172.21.15.110:0/1791936888 --> v2:172.21.15.110:6804/28368 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f013c040ca0 con 0x7f013c03e220 2024-09-15T04:05:08.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.943+0000 7f01537fe640 1 -- 172.21.15.110:0/1791936888 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=315) v2 ==== 24+0+0 (secure 0 0 0) 0x7f014c04dbc0 con 0x7f015c06e5a0 2024-09-15T04:05:08.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.943+0000 7f01537fe640 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-15T04:05:08.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.943+0000 7f01527fc640 1 --2- 172.21.15.110:0/1791936888 >> v2:172.21.15.177:6812/28201 conn(0x7f013c038700 0x7f013c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.943+0000 7f01527fc640 10 monclient: get_auth_request con 0x7f013c038700 auth_method 0 2024-09-15T04:05:08.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.943+0000 7f015b7fe640 1 --2- 172.21.15.110:0/1791936888 >> v2:172.21.15.110:6804/28368 conn(0x7f013c03e220 0x7f013c040630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:08.958 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.943+0000 7f015b7fe640 10 monclient: get_auth_request con 0x7f013c03e220 auth_method 0 2024-09-15T04:05:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.946+0000 7f01527fc640 1 --2- 172.21.15.110:0/1791936888 >> v2:172.21.15.177:6812/28201 conn(0x7f013c038700 0x7f013c03abb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0144034000 tx=0x7f0144032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.948+0000 7f015b7fe640 1 --2- 172.21.15.110:0/1791936888 >> v2:172.21.15.110:6804/28368 conn(0x7f013c03e220 0x7f013c040630 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:08.959 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.949+0000 7f01537fe640 1 -- 172.21.15.110:0/1791936888 <== osd.2 v2:172.21.15.110:6804/28368 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f013c040ca0 con 0x7f013c03e220 2024-09-15T04:05:08.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.959+0000 7f016184f640 1 -- 172.21.15.110:0/1791936888 --> v2:172.21.15.110:6804/28368 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f015c137070 con 0x7f013c03e220 2024-09-15T04:05:08.960 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.959+0000 7f01537fe640 1 -- 172.21.15.110:0/1791936888 <== osd.2 v2:172.21.15.110:6804/28368 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f015c137070 con 0x7f013c03e220 2024-09-15T04:05:08.961 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:05:08.961 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T04:05:08.961 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:05:08.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.960+0000 7f016184f640 1 -- 172.21.15.110:0/1791936888 >> v2:172.21.15.110:6804/28368 conn(0x7f013c03e220 msgr2=0x7f013c040630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.960+0000 7f016184f640 1 --2- 172.21.15.110:0/1791936888 >> v2:172.21.15.110:6804/28368 conn(0x7f013c03e220 0x7f013c040630 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.960+0000 7f016184f640 1 -- 172.21.15.110:0/1791936888 >> v2:172.21.15.177:6812/28201 conn(0x7f013c038700 msgr2=0x7f013c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.960+0000 7f016184f640 1 --2- 172.21.15.110:0/1791936888 >> v2:172.21.15.177:6812/28201 conn(0x7f013c038700 0x7f013c03abb0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0144034000 tx=0x7f0144032000 comp rx=0 tx=0).stop 2024-09-15T04:05:08.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.960+0000 7f016184f640 10 monclient: shutdown 2024-09-15T04:05:08.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.960+0000 7f016184f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:08.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.960+0000 7f016184f640 1 -- 172.21.15.110:0/1791936888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f015c06e5a0 msgr2=0x7f015c135e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:08.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.960+0000 7f016184f640 1 --2- 172.21.15.110:0/1791936888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f015c06e5a0 0x7f015c135e70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f014c00c5e0 tx=0x7f014c007ed0 comp rx=0 tx=0).stop 2024-09-15T04:05:08.962 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.960+0000 7f015b7fe640 1 -- 172.21.15.110:0/1791936888 reap_dead start 2024-09-15T04:05:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.960+0000 7f016184f640 1 -- 172.21.15.110:0/1791936888 shutdown_connections 2024-09-15T04:05:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.960+0000 7f016184f640 1 -- 172.21.15.110:0/1791936888 >> 172.21.15.110:0/1791936888 conn(0x7f015c0639f0 msgr2=0x7f015c0a2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.960+0000 7f016184f640 1 -- 172.21.15.110:0/1791936888 shutdown_connections 2024-09-15T04:05:08.963 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:08.960+0000 7f016184f640 1 -- 172.21.15.110:0/1791936888 wait complete. 2024-09-15T04:05:08.970 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-15T04:05:09.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.063+0000 7f01e3a4f640 10 monclient: get_monmap_and_config 2024-09-15T04:05:09.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.063+0000 7f01e3a4f640 10 monclient: build_initial_monmap 2024-09-15T04:05:09.064 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.063+0000 7f01e3a4f640 10 monclient: monmap: 2024-09-15T04:05:09.064 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:09.064 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.064 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:09.064356+0000 2024-09-15T04:05:09.064 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:09.064356+0000 2024-09-15T04:05:09.064 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:09.064 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.064 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:09.064 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:09.065 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:09.065 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:09.065 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:09.065 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.063+0000 7f01e3a4f640 1 Processor -- start 2024-09-15T04:05:09.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 1 -- start start 2024-09-15T04:05:09.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 10 monclient: init 2024-09-15T04:05:09.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:09.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-15T04:05:09.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01dc09f180 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:09.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 0x7f01dc0a0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 10 monclient(hunting): picked mon.noname-b con 0x7f01dc09fc60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:09.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f01dc1331f0 0x7f01dc1355d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 10 monclient(hunting): picked mon.noname-c con 0x7f01dc1331f0 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:09.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01dc0a05b0 con 0x7f01dc09f180 2024-09-15T04:05:09.067 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01dc06aaf0 con 0x7f01dc1331f0 2024-09-15T04:05:09.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc06d160 con 0x7f01dc09fc60 2024-09-15T04:05:09.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:09.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e3a4f640 10 monclient(hunting): authenticate will time out at 1132.708748s 2024-09-15T04:05:09.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e17eb640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f01dc09f180 0x7f01dc09f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:59002/0 (socket says 172.21.15.110:59002) 2024-09-15T04:05:09.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e17eb640 1 -- 172.21.15.110:0/4188303621 learned_addr learned my addr 172.21.15.110:0/4188303621 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:09.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e0fea640 1 --2- 172.21.15.110:0/4188303621 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 0x7f01dc0a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e0fea640 10 monclient(hunting): get_auth_request con 0x7f01dc09fc60 auth_method 0 2024-09-15T04:05:09.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e0fea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e0fea640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e0fea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.068 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e0fea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e0fea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.064+0000 7f01e0fea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.065+0000 7f01e1fec640 1 --2- 172.21.15.110:0/4188303621 >> v2:172.21.15.177:3300/0 conn(0x7f01dc1331f0 0x7f01dc1355d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.065+0000 7f01e0fea640 10 monclient(hunting): handle_auth_done global_id 74180 payload 274 2024-09-15T04:05:09.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.065+0000 7f01e0fea640 1 -- 172.21.15.110:0/4188303621 >> v2:172.21.15.177:3300/0 conn(0x7f01dc1331f0 msgr2=0x7f01dc1355d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:09.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.065+0000 7f01e0fea640 1 --2- 172.21.15.110:0/4188303621 >> v2:172.21.15.177:3300/0 conn(0x7f01dc1331f0 0x7f01dc1355d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.065+0000 7f01e0fea640 1 -- 172.21.15.110:0/4188303621 >> v1:172.21.15.110:6789/0 conn(0x7f01dc09f180 legacy=0x7f01dc09f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:09.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.065+0000 7f01e0fea640 10 monclient: _finish_hunting 0 2024-09-15T04:05:09.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.065+0000 7f01e0fea640 1 monclient: found mon.noname-b 2024-09-15T04:05:09.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.065+0000 7f01e0fea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:09.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.065+0000 7f01e0fea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.071 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.065+0000 7f01e0fea640 1 -- 172.21.15.110:0/4188303621 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01dc06acf0 con 0x7f01dc09fc60 2024-09-15T04:05:09.076 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.067+0000 7f01e0fea640 1 -- 172.21.15.110:0/4188303621 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 msgr2=0x7f01dc0a0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:09.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.067+0000 7f01e0fea640 1 --2- 172.21.15.110:0/4188303621 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 0x7f01dc0a0070 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01cc009880 tx=0x7f01cc0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd5cd7e6bf546d34 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01d3fff640 1 -- 172.21.15.110:0/4188303621 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc032440 con 0x7f01dc09fc60 2024-09-15T04:05:09.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:05:09.077 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01d3fff640 10 monclient: dump: 2024-09-15T04:05:09.077 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.077 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.077 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.077 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.077 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.077 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.077 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.077 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.078 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.078 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01d3fff640 10 monclient: _finish_auth 0 2024-09-15T04:05:09.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:09.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01e3a4f640 5 monclient: authenticate success, global_id 74180 2024-09-15T04:05:09.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01e3a4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:09.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01d3fff640 1 -- 172.21.15.110:0/4188303621 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01cc0325a0 con 0x7f01dc09fc60 2024-09-15T04:05:09.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01d3fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:09.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01e3a4f640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:09.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01e3a4f640 4 set_mon_vals no callback set 2024-09-15T04:05:09.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01d3fff640 1 -- 172.21.15.110:0/4188303621 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc032940 con 0x7f01dc09fc60 2024-09-15T04:05:09.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:09.078 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01d3fff640 10 monclient: dump: 2024-09-15T04:05:09.079 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.079 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.079 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.079 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.079 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.079 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.079 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.079 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.080 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.080 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01e3a4f640 10 monclient: shutdown 2024-09-15T04:05:09.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01e3a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:09.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01e3a4f640 1 -- 172.21.15.110:0/4188303621 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 msgr2=0x7f01dc0a0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01e3a4f640 1 --2- 172.21.15.110:0/4188303621 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 0x7f01dc0a0070 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f01cc009880 tx=0x7f01cc0090f0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01e3a4f640 1 -- 172.21.15.110:0/4188303621 shutdown_connections 2024-09-15T04:05:09.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01e3a4f640 1 --2- 172.21.15.110:0/4188303621 >> v2:172.21.15.177:3300/0 conn(0x7f01dc1331f0 0x7f01dc1355d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01e3a4f640 1 --2- 172.21.15.110:0/4188303621 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 0x7f01dc0a0070 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01e3a4f640 1 -- 172.21.15.110:0/4188303621 >> 172.21.15.110:0/4188303621 conn(0x7f01dc09d680 msgr2=0x7f01dc09da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:09.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01e3a4f640 1 -- 172.21.15.110:0/4188303621 shutdown_connections 2024-09-15T04:05:09.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.068+0000 7f01e3a4f640 1 -- 172.21.15.110:0/4188303621 wait complete. 2024-09-15T04:05:09.080 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.069+0000 7f01e3a4f640 10 monclient: build_initial_monmap 2024-09-15T04:05:09.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.069+0000 7f01e3a4f640 10 monclient: monmap: 2024-09-15T04:05:09.081 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:09.081 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:09.081 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:09.081 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:09.081 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:09.081 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.081 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:09.081 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:09.081 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:09.081 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.069+0000 7f01e3a4f640 1 Processor -- start 2024-09-15T04:05:09.081 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.069+0000 7f01e3a4f640 1 -- start start 2024-09-15T04:05:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.069+0000 7f01e3a4f640 10 monclient: init 2024-09-15T04:05:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T04:05:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01dc09f180 0x7f01dc131c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 10 monclient(hunting): picked mon.noname-a con 0x7f01dc09f180 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 0x7f01dc11b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 10 monclient(hunting): picked mon.noname-c con 0x7f01dc09fc60 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01dc1331f0 0x7f01dc11bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 10 monclient(hunting): picked mon.noname-b con 0x7f01dc1331f0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc11ef10 con 0x7f01dc09f180 2024-09-15T04:05:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc06b240 con 0x7f01dc1331f0 2024-09-15T04:05:09.082 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01dc099ca0 con 0x7f01dc09fc60 2024-09-15T04:05:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e3a4f640 10 monclient(hunting): authenticate will time out at 1132.714210s 2024-09-15T04:05:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 0x7f01dc11b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e17eb640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01dc09f180 0x7f01dc131c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 0x7f01dc11b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:35284/0 (socket says 172.21.15.110:35284) 2024-09-15T04:05:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 1 -- 172.21.15.110:0/3617802978 learned_addr learned my addr 172.21.15.110:0/3617802978 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 10 monclient(hunting): get_auth_request con 0x7f01dc09fc60 auth_method 0 2024-09-15T04:05:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e17eb640 10 monclient(hunting): get_auth_request con 0x7f01dc09f180 auth_method 0 2024-09-15T04:05:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e17eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.083 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e17eb640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e17eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e17eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e17eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e17eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 10 monclient(hunting): handle_auth_done global_id 74183 payload 824 2024-09-15T04:05:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 1 -- 172.21.15.110:0/3617802978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01dc1331f0 msgr2=0x7f01dc11bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 1 --2- 172.21.15.110:0/3617802978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01dc1331f0 0x7f01dc11bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 1 -- 172.21.15.110:0/3617802978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01dc09f180 msgr2=0x7f01dc131c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.084 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 1 --2- 172.21.15.110:0/3617802978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01dc09f180 0x7f01dc131c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 10 monclient: _finish_hunting 0 2024-09-15T04:05:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 1 monclient: found mon.noname-c 2024-09-15T04:05:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e0fea640 1 -- 172.21.15.110:0/3617802978 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc009390 con 0x7f01dc09fc60 2024-09-15T04:05:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.070+0000 7f01e17eb640 1 --2- 172.21.15.110:0/3617802978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01dc09f180 0x7f01dc131c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01e0fea640 1 -- 172.21.15.110:0/3617802978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 msgr2=0x7f01dc11b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01e0fea640 1 --2- 172.21.15.110:0/3617802978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 0x7f01dc11b5c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f01cc009360 tx=0x7f01cc032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01d17fa640 1 -- 172.21.15.110:0/3617802978 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc031c70 con 0x7f01dc09fc60 2024-09-15T04:05:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01d17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01d17fa640 10 monclient: dump: 2024-09-15T04:05:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.085 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.086 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01d17fa640 10 monclient: _finish_auth 0 2024-09-15T04:05:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01d17fa640 1 -- 172.21.15.110:0/3617802978 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01cc031dd0 con 0x7f01dc09fc60 2024-09-15T04:05:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01e3a4f640 5 monclient: authenticate success, global_id 74183 2024-09-15T04:05:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01e3a4f640 10 monclient: _renew_subs 2024-09-15T04:05:09.086 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01e3a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01e3a4f640 1 -- 172.21.15.110:0/3617802978 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01dc11d160 con 0x7f01dc09fc60 2024-09-15T04:05:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01d17fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01d17fa640 1 -- 172.21.15.110:0/3617802978 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc03d690 con 0x7f01dc09fc60 2024-09-15T04:05:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01d17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01d17fa640 10 monclient: dump: 2024-09-15T04:05:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.087 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.088 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.072+0000 7f01d37fe640 4 set_mon_vals no callback set 2024-09-15T04:05:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.073+0000 7f01e3a4f640 10 monclient: _renew_subs 2024-09-15T04:05:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.073+0000 7f01e3a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.073+0000 7f01e3a4f640 1 -- 172.21.15.110:0/3617802978 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01dc11f8f0 con 0x7f01dc09fc60 2024-09-15T04:05:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.073+0000 7f01d17fa640 1 -- 172.21.15.110:0/3617802978 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f01cc039040 con 0x7f01dc09fc60 2024-09-15T04:05:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.073+0000 7f01e3a4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.073+0000 7f01e3a4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.073+0000 7f01e3a4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.073+0000 7f01e3a4f640 1 -- 172.21.15.110:0/3617802978 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01dc11f150 con 0x7f01dc09fc60 2024-09-15T04:05:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.074+0000 7f01d17fa640 1 --2- 172.21.15.110:0/3617802978 >> v2:172.21.15.177:6812/28201 conn(0x7f01c0038700 0x7f01c003abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.074+0000 7f01d17fa640 1 -- 172.21.15.110:0/3617802978 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f01cc06b520 con 0x7f01dc09fc60 2024-09-15T04:05:09.088 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.074+0000 7f01e17eb640 1 --2- 172.21.15.110:0/3617802978 >> v2:172.21.15.177:6812/28201 conn(0x7f01c0038700 0x7f01c003abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.074+0000 7f01e17eb640 10 monclient: get_auth_request con 0x7f01c0038700 auth_method 0 2024-09-15T04:05:09.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.076+0000 7f01e17eb640 1 --2- 172.21.15.110:0/3617802978 >> v2:172.21.15.177:6812/28201 conn(0x7f01c0038700 0x7f01c003abb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f01d8009e40 tx=0x7f01d8006f50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.078+0000 7f01d17fa640 1 -- 172.21.15.110:0/3617802978 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f01cc02f4d0 con 0x7f01dc09fc60 2024-09-15T04:05:09.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.078+0000 7f01d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:09.089 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.078+0000 7f01d17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:09.235 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.234+0000 7f01e3a4f640 1 -- 172.21.15.110:0/3617802978 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f01dc0647e0 con 0x7f01c0038700 2024-09-15T04:05:09.236 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.234+0000 7f01d17fa640 1 -- 172.21.15.110:0/3617802978 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f01dc0647e0 con 0x7f01c0038700 2024-09-15T04:05:09.236 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 2 to deep-scrub 2024-09-15T04:05:09.237 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.236+0000 7f01e3a4f640 1 -- 172.21.15.110:0/3617802978 >> v2:172.21.15.177:6812/28201 conn(0x7f01c0038700 msgr2=0x7f01c003abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.236+0000 7f01e3a4f640 1 --2- 172.21.15.110:0/3617802978 >> v2:172.21.15.177:6812/28201 conn(0x7f01c0038700 0x7f01c003abb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f01d8009e40 tx=0x7f01d8006f50 comp rx=0 tx=0).stop 2024-09-15T04:05:09.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.236+0000 7f01e3a4f640 10 monclient: shutdown 2024-09-15T04:05:09.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.236+0000 7f01e3a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:09.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.236+0000 7f01e3a4f640 1 -- 172.21.15.110:0/3617802978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 msgr2=0x7f01dc11b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.236+0000 7f01e3a4f640 1 --2- 172.21.15.110:0/3617802978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 0x7f01dc11b5c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f01cc009360 tx=0x7f01cc032000 comp rx=0 tx=0).stop 2024-09-15T04:05:09.238 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.236+0000 7f01e3a4f640 1 -- 172.21.15.110:0/3617802978 shutdown_connections 2024-09-15T04:05:09.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.236+0000 7f01e3a4f640 1 --2- 172.21.15.110:0/3617802978 >> v2:172.21.15.177:6812/28201 conn(0x7f01c0038700 0x7f01c003abb0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.236+0000 7f01e3a4f640 1 --2- 172.21.15.110:0/3617802978 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f01dc1331f0 0x7f01dc11bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.236+0000 7f01e3a4f640 1 --2- 172.21.15.110:0/3617802978 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f01dc09fc60 0x7f01dc11b5c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.236+0000 7f01e3a4f640 1 --2- 172.21.15.110:0/3617802978 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01dc09f180 0x7f01dc131c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.236+0000 7f01e3a4f640 1 -- 172.21.15.110:0/3617802978 >> 172.21.15.110:0/3617802978 conn(0x7f01dc09d680 msgr2=0x7f01dc099fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:09.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.236+0000 7f01e3a4f640 1 -- 172.21.15.110:0/3617802978 shutdown_connections 2024-09-15T04:05:09.239 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.236+0000 7f01e3a4f640 1 -- 172.21.15.110:0/3617802978 wait complete. 2024-09-15T04:05:09.247 INFO:tasks.ceph:Scrubbing osd.3 2024-09-15T04:05:09.248 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-15T04:05:09.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.343+0000 7f82ae0af640 10 monclient: get_monmap_and_config 2024-09-15T04:05:09.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.343+0000 7f82ae0af640 10 monclient: build_initial_monmap 2024-09-15T04:05:09.344 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.343+0000 7f82ae0af640 10 monclient: monmap: 2024-09-15T04:05:09.344 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:09.344 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:09.344572+0000 2024-09-15T04:05:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:09.344572+0000 2024-09-15T04:05:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:09.345 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:09.345 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 1 Processor -- start 2024-09-15T04:05:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 1 -- start start 2024-09-15T04:05:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 10 monclient: init 2024-09-15T04:05:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T04:05:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f82a8131630 0x7f82a8131a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 10 monclient(hunting): picked mon.noname-a con 0x7f82a8131630 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82a8131f40 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:09.348 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 0x7f82a8138970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 10 monclient(hunting): picked mon.noname-b con 0x7f82a806ecb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82a809f320 con 0x7f82a8131f40 2024-09-15T04:05:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82a806f130 con 0x7f82a8131630 2024-09-15T04:05:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a8132310 con 0x7f82a806ecb0 2024-09-15T04:05:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82ae0af640 10 monclient(hunting): authenticate will time out at 1132.988813s 2024-09-15T04:05:09.349 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82a77fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f82a8131630 0x7f82a8131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82a7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 0x7f82a8138970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82a7fff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 0x7f82a8138970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:37156/0 (socket says 172.21.15.110:37156) 2024-09-15T04:05:09.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82a7fff640 1 -- 172.21.15.110:0/4166047648 learned_addr learned my addr 172.21.15.110:0/4166047648 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:09.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82a77fe640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f82a8131630 0x7f82a8131a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54624/0 (socket says 172.21.15.110:54624) 2024-09-15T04:05:09.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82a7fff640 10 monclient(hunting): get_auth_request con 0x7f82a806ecb0 auth_method 0 2024-09-15T04:05:09.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.344+0000 7f82a77fe640 10 monclient(hunting): get_auth_request con 0x7f82a8131630 auth_method 0 2024-09-15T04:05:09.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.350 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a7fff640 10 monclient(hunting): handle_auth_done global_id 74186 payload 274 2024-09-15T04:05:09.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a7fff640 1 -- 172.21.15.110:0/4166047648 >> v2:172.21.15.110:3300/0 conn(0x7f82a8131630 msgr2=0x7f82a8131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:09.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a7fff640 1 --2- 172.21.15.110:0/4166047648 >> v2:172.21.15.110:3300/0 conn(0x7f82a8131630 0x7f82a8131a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a7fff640 1 -- 172.21.15.110:0/4166047648 >> v1:172.21.15.177:6789/0 conn(0x7f82a8131f40 legacy=0x7f82a806e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:09.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a7fff640 10 monclient: _finish_hunting 0 2024-09-15T04:05:09.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a7fff640 1 monclient: found mon.noname-b 2024-09-15T04:05:09.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:09.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.351 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a7fff640 1 -- 172.21.15.110:0/4166047648 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a8068390 con 0x7f82a806ecb0 2024-09-15T04:05:09.352 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.345+0000 7f82a77fe640 1 --2- 172.21.15.110:0/4166047648 >> v2:172.21.15.110:3300/0 conn(0x7f82a8131630 0x7f82a8131a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:09.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.347+0000 7f82a7fff640 1 -- 172.21.15.110:0/4166047648 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 msgr2=0x7f82a8138970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:09.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.347+0000 7f82a7fff640 1 --2- 172.21.15.110:0/4166047648 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 0x7f82a8138970 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009f20 tx=0x7f82a0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=592e6ce4092587fb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82a67fc640 1 -- 172.21.15.110:0/4166047648 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a0007610 con 0x7f82a806ecb0 2024-09-15T04:05:09.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82a67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:05:09.358 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82a67fc640 10 monclient: dump: 2024-09-15T04:05:09.359 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.359 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.359 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.359 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.359 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.359 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.359 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.359 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.360 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82a67fc640 10 monclient: _finish_auth 0 2024-09-15T04:05:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82ae0af640 5 monclient: authenticate success, global_id 74186 2024-09-15T04:05:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82a67fc640 1 -- 172.21.15.110:0/4166047648 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a002d050 con 0x7f82a806ecb0 2024-09-15T04:05:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82ae0af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82a67fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82ae0af640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82ae0af640 4 set_mon_vals no callback set 2024-09-15T04:05:09.360 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82a67fc640 1 -- 172.21.15.110:0/4166047648 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a00079a0 con 0x7f82a806ecb0 2024-09-15T04:05:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82a67fc640 10 monclient: dump: 2024-09-15T04:05:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.361 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.362 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82ae0af640 10 monclient: shutdown 2024-09-15T04:05:09.362 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82ae0af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:09.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82ae0af640 1 -- 172.21.15.110:0/4166047648 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 msgr2=0x7f82a8138970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82ae0af640 1 --2- 172.21.15.110:0/4166047648 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 0x7f82a8138970 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009f20 tx=0x7f82a0009840 comp rx=0 tx=0).stop 2024-09-15T04:05:09.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82ae0af640 1 -- 172.21.15.110:0/4166047648 shutdown_connections 2024-09-15T04:05:09.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82ae0af640 1 --2- 172.21.15.110:0/4166047648 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 0x7f82a8138970 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82ae0af640 1 --2- 172.21.15.110:0/4166047648 >> v2:172.21.15.110:3300/0 conn(0x7f82a8131630 0x7f82a8131a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82ae0af640 1 -- 172.21.15.110:0/4166047648 >> 172.21.15.110:0/4166047648 conn(0x7f82a80639f0 msgr2=0x7f82a80a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:09.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82ae0af640 1 -- 172.21.15.110:0/4166047648 shutdown_connections 2024-09-15T04:05:09.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.348+0000 7f82ae0af640 1 -- 172.21.15.110:0/4166047648 wait complete. 2024-09-15T04:05:09.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.349+0000 7f82ae0af640 10 monclient: build_initial_monmap 2024-09-15T04:05:09.363 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.349+0000 7f82ae0af640 10 monclient: monmap: 2024-09-15T04:05:09.363 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:09.363 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:09.364 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:09.364 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:09.364 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:09.364 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.364 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:09.364 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:09.364 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:09.364 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.349+0000 7f82ae0af640 1 Processor -- start 2024-09-15T04:05:09.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.349+0000 7f82ae0af640 1 -- start start 2024-09-15T04:05:09.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.349+0000 7f82ae0af640 10 monclient: init 2024-09-15T04:05:09.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:09.364 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T04:05:09.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 0x7f82a812eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 10 monclient(hunting): picked mon.noname-c con 0x7f82a806ecb0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:09.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82a8131630 0x7f82a8133ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 10 monclient(hunting): picked mon.noname-b con 0x7f82a8131630 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:09.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82a8131f40 0x7f82a81340e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 10 monclient(hunting): picked mon.noname-a con 0x7f82a8131f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:09.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a8135bd0 con 0x7f82a8131f40 2024-09-15T04:05:09.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82a80688c0 con 0x7f82a8131630 2024-09-15T04:05:09.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.365 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82a81388b0 con 0x7f82a806ecb0 2024-09-15T04:05:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82ae0af640 10 monclient(hunting): authenticate will time out at 1132.994235s 2024-09-15T04:05:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 0x7f82a812eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 0x7f82a812eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:37168/0 (socket says 172.21.15.110:37168) 2024-09-15T04:05:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 1 -- 172.21.15.110:0/27531914 learned_addr learned my addr 172.21.15.110:0/27531914 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 10 monclient(hunting): get_auth_request con 0x7f82a806ecb0 auth_method 0 2024-09-15T04:05:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a7fff640 1 --2- 172.21.15.110:0/27531914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82a8131f40 0x7f82a81340e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a7fff640 10 monclient(hunting): get_auth_request con 0x7f82a8131f40 auth_method 0 2024-09-15T04:05:09.366 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 10 monclient(hunting): handle_auth_done global_id 74189 payload 824 2024-09-15T04:05:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a6ffd640 1 --2- 172.21.15.110:0/27531914 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82a8131630 0x7f82a8133ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 1 -- 172.21.15.110:0/27531914 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82a8131630 msgr2=0x7f82a8133ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 1 --2- 172.21.15.110:0/27531914 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82a8131630 0x7f82a8133ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 1 -- 172.21.15.110:0/27531914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82a8131f40 msgr2=0x7f82a81340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.367 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 1 --2- 172.21.15.110:0/27531914 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82a8131f40 0x7f82a81340e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 10 monclient: _finish_hunting 0 2024-09-15T04:05:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 1 monclient: found mon.noname-c 2024-09-15T04:05:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a77fe640 1 -- 172.21.15.110:0/27531914 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c009390 con 0x7f82a806ecb0 2024-09-15T04:05:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a6ffd640 10 monclient: get_auth_request con 0x7f82a8131630 auth_method 0 2024-09-15T04:05:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.350+0000 7f82a6ffd640 1 --2- 172.21.15.110:0/27531914 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82a8131630 0x7f82a8133ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:05:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.352+0000 7f82a77fe640 1 -- 172.21.15.110:0/27531914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 msgr2=0x7f82a812eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.352+0000 7f82a77fe640 1 --2- 172.21.15.110:0/27531914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 0x7f82a812eec0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f829c00ec80 tx=0x7f829c0090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.352+0000 7f828f7fe640 1 -- 172.21.15.110:0/27531914 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829c00eec0 con 0x7f82a806ecb0 2024-09-15T04:05:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.352+0000 7f828f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.352+0000 7f828f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.352+0000 7f828f7fe640 10 monclient: dump: 2024-09-15T04:05:09.368 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.369 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.369 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.369 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.369 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.369 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.369 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.369 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.369 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.369 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.352+0000 7f828f7fe640 10 monclient: _finish_auth 0 2024-09-15T04:05:09.369 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.352+0000 7f828f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:09.370 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.352+0000 7f828f7fe640 1 -- 172.21.15.110:0/27531914 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f829c00cdd0 con 0x7f82a806ecb0 2024-09-15T04:05:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.352+0000 7f82ae0af640 5 monclient: authenticate success, global_id 74189 2024-09-15T04:05:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f82ae0af640 10 monclient: _renew_subs 2024-09-15T04:05:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f82ae0af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f82ae0af640 1 -- 172.21.15.110:0/27531914 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82a0009ae0 con 0x7f82a806ecb0 2024-09-15T04:05:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f828f7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f828f7fe640 1 -- 172.21.15.110:0/27531914 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829c013680 con 0x7f82a806ecb0 2024-09-15T04:05:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f828f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f828f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f828f7fe640 10 monclient: dump: 2024-09-15T04:05:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.371 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.372 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f82ae0af640 10 monclient: _renew_subs 2024-09-15T04:05:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f82a5ffb640 4 set_mon_vals no callback set 2024-09-15T04:05:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f82ae0af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f82ae0af640 1 -- 172.21.15.110:0/27531914 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82a8137230 con 0x7f82a806ecb0 2024-09-15T04:05:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f82ae0af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f82ae0af640 1 -- 172.21.15.110:0/27531914 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f82a8070950 con 0x7f82a806ecb0 2024-09-15T04:05:09.372 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.353+0000 7f828f7fe640 1 -- 172.21.15.110:0/27531914 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f829c0139f0 con 0x7f82a806ecb0 2024-09-15T04:05:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.354+0000 7f828f7fe640 1 --2- 172.21.15.110:0/27531914 >> v2:172.21.15.177:6812/28201 conn(0x7f8294038730 0x7f829403abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.354+0000 7f828f7fe640 1 -- 172.21.15.110:0/27531914 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f829c04b970 con 0x7f82a806ecb0 2024-09-15T04:05:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.354+0000 7f828f7fe640 1 --2- 172.21.15.110:0/27531914 >> v2:172.21.15.177:6804/28628 conn(0x7f829403e250 0x7f8294040660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.354+0000 7f828f7fe640 1 -- 172.21.15.110:0/27531914 --> v2:172.21.15.177:6804/28628 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8294040cd0 con 0x7f829403e250 2024-09-15T04:05:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.354+0000 7f828f7fe640 1 -- 172.21.15.110:0/27531914 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=315) v2 ==== 24+0+0 (secure 0 0 0) 0x7f829c0502a0 con 0x7f82a806ecb0 2024-09-15T04:05:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.354+0000 7f828f7fe640 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-15T04:05:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.354+0000 7f82a6ffd640 1 --2- 172.21.15.110:0/27531914 >> v2:172.21.15.177:6812/28201 conn(0x7f8294038730 0x7f829403abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.354+0000 7f82a6ffd640 10 monclient: get_auth_request con 0x7f8294038730 auth_method 0 2024-09-15T04:05:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.354+0000 7f82a7fff640 1 --2- 172.21.15.110:0/27531914 >> v2:172.21.15.177:6804/28628 conn(0x7f829403e250 0x7f8294040660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.354+0000 7f82a7fff640 10 monclient: get_auth_request con 0x7f829403e250 auth_method 0 2024-09-15T04:05:09.373 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.357+0000 7f82a6ffd640 1 --2- 172.21.15.110:0/27531914 >> v2:172.21.15.177:6812/28201 conn(0x7f8294038730 0x7f829403abe0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f82a8131f10 tx=0x7f8298009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.374 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.359+0000 7f82a7fff640 1 --2- 172.21.15.110:0/27531914 >> v2:172.21.15.177:6804/28628 conn(0x7f829403e250 0x7f8294040660 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.360+0000 7f828f7fe640 1 -- 172.21.15.110:0/27531914 <== osd.3 v2:172.21.15.177:6804/28628 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f8294040cd0 con 0x7f829403e250 2024-09-15T04:05:09.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.370+0000 7f82ae0af640 1 -- 172.21.15.110:0/27531914 --> v2:172.21.15.177:6804/28628 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f82a8138100 con 0x7f829403e250 2024-09-15T04:05:09.375 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.370+0000 7f828f7fe640 1 -- 172.21.15.110:0/27531914 <== osd.3 v2:172.21.15.177:6804/28628 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f82a8138100 con 0x7f829403e250 2024-09-15T04:05:09.375 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:05:09.375 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T04:05:09.375 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:05:09.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.371+0000 7f82ae0af640 1 -- 172.21.15.110:0/27531914 >> v2:172.21.15.177:6804/28628 conn(0x7f829403e250 msgr2=0x7f8294040660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.376 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.371+0000 7f82ae0af640 1 --2- 172.21.15.110:0/27531914 >> v2:172.21.15.177:6804/28628 conn(0x7f829403e250 0x7f8294040660 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.371+0000 7f82ae0af640 1 -- 172.21.15.110:0/27531914 >> v2:172.21.15.177:6812/28201 conn(0x7f8294038730 msgr2=0x7f829403abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.371+0000 7f82ae0af640 1 --2- 172.21.15.110:0/27531914 >> v2:172.21.15.177:6812/28201 conn(0x7f8294038730 0x7f829403abe0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f82a8131f10 tx=0x7f8298009040 comp rx=0 tx=0).stop 2024-09-15T04:05:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.371+0000 7f82ae0af640 10 monclient: shutdown 2024-09-15T04:05:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.371+0000 7f82ae0af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.371+0000 7f82ae0af640 1 -- 172.21.15.110:0/27531914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 msgr2=0x7f82a812eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.371+0000 7f82ae0af640 1 --2- 172.21.15.110:0/27531914 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f82a806ecb0 0x7f82a812eec0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f829c00ec80 tx=0x7f829c0090f0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.371+0000 7f82a7fff640 1 -- 172.21.15.110:0/27531914 reap_dead start 2024-09-15T04:05:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.371+0000 7f82ae0af640 1 -- 172.21.15.110:0/27531914 shutdown_connections 2024-09-15T04:05:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.371+0000 7f82ae0af640 1 -- 172.21.15.110:0/27531914 >> 172.21.15.110:0/27531914 conn(0x7f82a80639f0 msgr2=0x7f82a8130af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.371+0000 7f82ae0af640 1 -- 172.21.15.110:0/27531914 shutdown_connections 2024-09-15T04:05:09.377 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.371+0000 7f82ae0af640 1 -- 172.21.15.110:0/27531914 wait complete. 2024-09-15T04:05:09.382 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-15T04:05:09.476 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.476+0000 7f084a19d640 10 monclient: get_monmap_and_config 2024-09-15T04:05:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.476+0000 7f084a19d640 10 monclient: build_initial_monmap 2024-09-15T04:05:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.476+0000 7f084a19d640 10 monclient: monmap: 2024-09-15T04:05:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:09.477116+0000 2024-09-15T04:05:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:09.477116+0000 2024-09-15T04:05:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:09.477 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:09.478 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:09.478 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.476+0000 7f084a19d640 1 Processor -- start 2024-09-15T04:05:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.476+0000 7f084a19d640 1 -- start start 2024-09-15T04:05:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.476+0000 7f084a19d640 10 monclient: init 2024-09-15T04:05:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f084a19d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f084a19d640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-15T04:05:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f084a19d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f08440a1a10 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f084a19d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 0x7f0844131b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f084a19d640 10 monclient(hunting): picked mon.noname-b con 0x7f084409ee80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f084a19d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08441369e0 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f084a19d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084409f290 con 0x7f08441369e0 2024-09-15T04:05:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f084a19d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08440a0850 con 0x7f08440a1a10 2024-09-15T04:05:09.480 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f084a19d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f084a19d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084409c9a0 con 0x7f084409ee80 2024-09-15T04:05:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f084a19d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f084a19d640 10 monclient(hunting): authenticate will time out at 1133.121504s 2024-09-15T04:05:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 0x7f0844131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 0x7f0844131b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:37170/0 (socket says 172.21.15.110:37170) 2024-09-15T04:05:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 1 -- 172.21.15.110:0/1164436508 learned_addr learned my addr 172.21.15.110:0/1164436508 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 10 monclient(hunting): get_auth_request con 0x7f084409ee80 auth_method 0 2024-09-15T04:05:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0843fff640 1 --1- 172.21.15.110:0/1164436508 >> v1:172.21.15.110:6789/0 conn(0x7f08441369e0 0x7f0844138dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51484/0 (socket says 172.21.15.110:51484) 2024-09-15T04:05:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.481 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 10 monclient(hunting): handle_auth_done global_id 74192 payload 274 2024-09-15T04:05:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 1 -- 172.21.15.110:0/1164436508 >> v1:172.21.15.177:6789/0 conn(0x7f08440a1a10 legacy=0x7f084409e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 1 -- 172.21.15.110:0/1164436508 >> v1:172.21.15.110:6789/0 conn(0x7f08441369e0 legacy=0x7f0844138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 10 monclient: _finish_hunting 0 2024-09-15T04:05:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 1 monclient: found mon.noname-b 2024-09-15T04:05:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.482 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.477+0000 7f0842ffd640 1 -- 172.21.15.110:0/1164436508 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084409cc80 con 0x7f084409ee80 2024-09-15T04:05:09.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f0842ffd640 1 -- 172.21.15.110:0/1164436508 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 msgr2=0x7f0844131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:09.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f0842ffd640 1 --2- 172.21.15.110:0/1164436508 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 0x7f0844131b90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f082c009940 tx=0x7f082c0091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=623fa01df5550e0c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f08427fc640 1 -- 172.21.15.110:0/1164436508 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c032440 con 0x7f084409ee80 2024-09-15T04:05:09.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f08427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f08427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:05:09.488 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f08427fc640 10 monclient: dump: 2024-09-15T04:05:09.488 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.489 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.489 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.489 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.489 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.489 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.489 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.489 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.489 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.489 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f08427fc640 10 monclient: _finish_auth 0 2024-09-15T04:05:09.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f08427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:09.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f08427fc640 1 -- 172.21.15.110:0/1164436508 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c0325a0 con 0x7f084409ee80 2024-09-15T04:05:09.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f084a19d640 5 monclient: authenticate success, global_id 74192 2024-09-15T04:05:09.489 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f084a19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f08427fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f084a19d640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f08427fc640 1 -- 172.21.15.110:0/1164436508 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c032940 con 0x7f084409ee80 2024-09-15T04:05:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f08427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f084a19d640 4 set_mon_vals no callback set 2024-09-15T04:05:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f08427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.480+0000 7f08427fc640 10 monclient: dump: 2024-09-15T04:05:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.490 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.491 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.481+0000 7f084a19d640 10 monclient: shutdown 2024-09-15T04:05:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.481+0000 7f084a19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.481+0000 7f084a19d640 1 -- 172.21.15.110:0/1164436508 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 msgr2=0x7f0844131b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.481+0000 7f084a19d640 1 --2- 172.21.15.110:0/1164436508 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 0x7f0844131b90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f082c009940 tx=0x7f082c0091d0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.481+0000 7f084a19d640 1 -- 172.21.15.110:0/1164436508 shutdown_connections 2024-09-15T04:05:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.481+0000 7f084a19d640 1 --2- 172.21.15.110:0/1164436508 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 0x7f0844131b90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.481+0000 7f084a19d640 1 -- 172.21.15.110:0/1164436508 >> 172.21.15.110:0/1164436508 conn(0x7f084406e860 msgr2=0x7f084406ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.481+0000 7f084a19d640 1 -- 172.21.15.110:0/1164436508 shutdown_connections 2024-09-15T04:05:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.481+0000 7f084a19d640 1 -- 172.21.15.110:0/1164436508 wait complete. 2024-09-15T04:05:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.481+0000 7f084a19d640 10 monclient: build_initial_monmap 2024-09-15T04:05:09.491 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.481+0000 7f084a19d640 10 monclient: monmap: 2024-09-15T04:05:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:09.492 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 1 Processor -- start 2024-09-15T04:05:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 1 -- start start 2024-09-15T04:05:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 10 monclient: init 2024-09-15T04:05:09.492 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 0x7f084409b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 10 monclient(hunting): picked mon.noname-c con 0x7f084409ee80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08440a1a10 0x7f0844096090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 10 monclient(hunting): picked mon.noname-a con 0x7f08440a1a10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08441369e0 0x7f08440965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 10 monclient(hunting): picked mon.noname-b con 0x7f08441369e0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08440a2d20 con 0x7f08440a1a10 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08440996e0 con 0x7f08441369e0 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08440999c0 con 0x7f084409ee80 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:09.493 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.482+0000 7f084a19d640 10 monclient(hunting): authenticate will time out at 1133.126999s 2024-09-15T04:05:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f08437fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 0x7f084409b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f08437fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 0x7f084409b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:37184/0 (socket says 172.21.15.110:37184) 2024-09-15T04:05:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08440a1a10 0x7f0844096090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f08437fe640 1 -- 172.21.15.110:0/53628640 learned_addr learned my addr 172.21.15.110:0/53628640 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f08437fe640 10 monclient(hunting): get_auth_request con 0x7f084409ee80 auth_method 0 2024-09-15T04:05:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f08437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f08437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f08437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 10 monclient(hunting): get_auth_request con 0x7f08440a1a10 auth_method 0 2024-09-15T04:05:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.494 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f08437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f08437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f08437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 10 monclient(hunting): handle_auth_done global_id 55242 payload 824 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 1 -- 172.21.15.110:0/53628640 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 msgr2=0x7f084409b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 1 --2- 172.21.15.110:0/53628640 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 0x7f084409b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 1 -- 172.21.15.110:0/53628640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08441369e0 msgr2=0x7f08440965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0843fff640 1 --2- 172.21.15.110:0/53628640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08441369e0 0x7f08440965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 1 --2- 172.21.15.110:0/53628640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08441369e0 0x7f08440965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 10 monclient: _finish_hunting 0 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 1 monclient: found mon.noname-a 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:09.495 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0842ffd640 1 -- 172.21.15.110:0/53628640 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c009450 con 0x7f08440a1a10 2024-09-15T04:05:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0843fff640 10 monclient: get_auth_request con 0x7f08441369e0 auth_method 0 2024-09-15T04:05:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f08437fe640 1 --2- 172.21.15.110:0/53628640 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 0x7f084409b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.483+0000 7f0843fff640 1 --2- 172.21.15.110:0/53628640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08441369e0 0x7f08440965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T04:05:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f0842ffd640 1 -- 172.21.15.110:0/53628640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08440a1a10 msgr2=0x7f0844096090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f0842ffd640 1 --2- 172.21.15.110:0/53628640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08440a1a10 0x7f0844096090 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f082c031000 tx=0x7f082c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f083b7fe640 1 -- 172.21.15.110:0/53628640 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c03c960 con 0x7f08440a1a10 2024-09-15T04:05:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f083b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f083b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f083b7fe640 10 monclient: dump: 2024-09-15T04:05:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.496 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f083b7fe640 10 monclient: _finish_auth 0 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f083b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f084a19d640 5 monclient: authenticate success, global_id 55242 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f083b7fe640 1 -- 172.21.15.110:0/53628640 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c03cac0 con 0x7f08440a1a10 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f084a19d640 10 monclient: _renew_subs 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f084a19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f084a19d640 1 -- 172.21.15.110:0/53628640 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08440987f0 con 0x7f08440a1a10 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f083b7fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f083b7fe640 1 -- 172.21.15.110:0/53628640 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082c03ce60 con 0x7f08440a1a10 2024-09-15T04:05:09.497 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f083b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f083b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f083b7fe640 10 monclient: dump: 2024-09-15T04:05:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.498 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f0841ffb640 4 set_mon_vals no callback set 2024-09-15T04:05:09.498 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f084a19d640 10 monclient: _renew_subs 2024-09-15T04:05:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f084a19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.485+0000 7f084a19d640 1 -- 172.21.15.110:0/53628640 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084409a540 con 0x7f08440a1a10 2024-09-15T04:05:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.486+0000 7f083b7fe640 1 -- 172.21.15.110:0/53628640 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f082c039040 con 0x7f08440a1a10 2024-09-15T04:05:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.486+0000 7f084a19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.486+0000 7f084a19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.486+0000 7f084a19d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.486+0000 7f084a19d640 1 -- 172.21.15.110:0/53628640 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08440a1230 con 0x7f08440a1a10 2024-09-15T04:05:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.486+0000 7f083b7fe640 1 --2- 172.21.15.110:0/53628640 >> v2:172.21.15.177:6812/28201 conn(0x7f0824038700 0x7f082403abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.486+0000 7f083b7fe640 1 -- 172.21.15.110:0/53628640 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f082c06b670 con 0x7f08440a1a10 2024-09-15T04:05:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.487+0000 7f08437fe640 1 --2- 172.21.15.110:0/53628640 >> v2:172.21.15.177:6812/28201 conn(0x7f0824038700 0x7f082403abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.499 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.487+0000 7f08437fe640 10 monclient: get_auth_request con 0x7f0824038700 auth_method 0 2024-09-15T04:05:09.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.489+0000 7f08437fe640 1 --2- 172.21.15.110:0/53628640 >> v2:172.21.15.177:6812/28201 conn(0x7f0824038700 0x7f082403abb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f083400f470 tx=0x7f08340026c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.491+0000 7f083b7fe640 1 -- 172.21.15.110:0/53628640 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f08440a1230 con 0x7f08440a1a10 2024-09-15T04:05:09.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.491+0000 7f083b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:09.500 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.491+0000 7f083b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:09.644 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.643+0000 7f084a19d640 1 -- 172.21.15.110:0/53628640 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f08440a1230 con 0x7f0824038700 2024-09-15T04:05:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.644+0000 7f083b7fe640 1 -- 172.21.15.110:0/53628640 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f08440a1230 con 0x7f0824038700 2024-09-15T04:05:09.645 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 3 to deep-scrub 2024-09-15T04:05:09.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.646+0000 7f084a19d640 1 -- 172.21.15.110:0/53628640 >> v2:172.21.15.177:6812/28201 conn(0x7f0824038700 msgr2=0x7f082403abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.646+0000 7f084a19d640 1 --2- 172.21.15.110:0/53628640 >> v2:172.21.15.177:6812/28201 conn(0x7f0824038700 0x7f082403abb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f083400f470 tx=0x7f08340026c0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.646+0000 7f084a19d640 10 monclient: shutdown 2024-09-15T04:05:09.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.646+0000 7f084a19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:09.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.646+0000 7f084a19d640 1 -- 172.21.15.110:0/53628640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08440a1a10 msgr2=0x7f0844096090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.646+0000 7f084a19d640 1 --2- 172.21.15.110:0/53628640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08440a1a10 0x7f0844096090 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f082c031000 tx=0x7f082c032000 comp rx=0 tx=0).stop 2024-09-15T04:05:09.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.646+0000 7f084a19d640 1 -- 172.21.15.110:0/53628640 shutdown_connections 2024-09-15T04:05:09.647 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.646+0000 7f084a19d640 1 --2- 172.21.15.110:0/53628640 >> v2:172.21.15.177:6812/28201 conn(0x7f0824038700 0x7f082403abb0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.646+0000 7f084a19d640 1 --2- 172.21.15.110:0/53628640 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f08441369e0 0x7f08440965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.646+0000 7f084a19d640 1 --2- 172.21.15.110:0/53628640 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f08440a1a10 0x7f0844096090 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.646+0000 7f084a19d640 1 --2- 172.21.15.110:0/53628640 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f084409ee80 0x7f084409b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.646+0000 7f084a19d640 1 -- 172.21.15.110:0/53628640 >> 172.21.15.110:0/53628640 conn(0x7f084406e860 msgr2=0x7f0844138dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:09.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.646+0000 7f084a19d640 1 -- 172.21.15.110:0/53628640 shutdown_connections 2024-09-15T04:05:09.648 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.646+0000 7f084a19d640 1 -- 172.21.15.110:0/53628640 wait complete. 2024-09-15T04:05:09.657 INFO:tasks.ceph:Scrubbing osd.4 2024-09-15T04:05:09.657 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-15T04:05:09.753 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.752+0000 7f5755b5d640 10 monclient: get_monmap_and_config 2024-09-15T04:05:09.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.752+0000 7f5755b5d640 10 monclient: build_initial_monmap 2024-09-15T04:05:09.754 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.752+0000 7f5755b5d640 10 monclient: monmap: 2024-09-15T04:05:09.754 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:09.754 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.754 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:09.753896+0000 2024-09-15T04:05:09.754 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:09.753896+0000 2024-09-15T04:05:09.754 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:09.754 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.754 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:09.754 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:09.754 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:09.754 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:09.754 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:09.754 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.753+0000 7f5755b5d640 1 Processor -- start 2024-09-15T04:05:09.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.753+0000 7f5755b5d640 1 -- start start 2024-09-15T04:05:09.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.753+0000 7f5755b5d640 10 monclient: init 2024-09-15T04:05:09.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f5755b5d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:09.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f5755b5d640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-15T04:05:09.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f5755b5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5750067130 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:09.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f5755b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5750067500 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:09.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f5755b5d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5750135930 0x7f5750137d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f5755b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f5750135930 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:09.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f5755b5d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575009c670 con 0x7f5750067500 2024-09-15T04:05:09.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f5755b5d640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575009cd70 con 0x7f5750067130 2024-09-15T04:05:09.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f5755b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.757 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f5755b5d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575009ca90 con 0x7f5750135930 2024-09-15T04:05:09.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f5755b5d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:09.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f5755b5d640 10 monclient(hunting): authenticate will time out at 1133.398226s 2024-09-15T04:05:09.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5750135930 0x7f5750137d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574effd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f5750067500 0x7f5750067900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51492/0 (socket says 172.21.15.110:51492) 2024-09-15T04:05:09.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574effd640 1 -- 172.21.15.110:0/1144812383 learned_addr learned my addr 172.21.15.110:0/1144812383 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:09.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 10 monclient(hunting): get_auth_request con 0x7f5750135930 auth_method 0 2024-09-15T04:05:09.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 10 monclient(hunting): handle_auth_done global_id 55245 payload 274 2024-09-15T04:05:09.758 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 1 -- 172.21.15.110:0/1144812383 >> v1:172.21.15.177:6789/0 conn(0x7f5750067130 legacy=0x7f575009dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:09.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 1 -- 172.21.15.110:0/1144812383 >> v1:172.21.15.110:6789/0 conn(0x7f5750067500 legacy=0x7f5750067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:09.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 10 monclient: _finish_hunting 0 2024-09-15T04:05:09.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 1 monclient: found mon.noname-a 2024-09-15T04:05:09.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:09.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:09.759 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.754+0000 7f574ffff640 1 -- 172.21.15.110:0/1144812383 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575009d050 con 0x7f5750135930 2024-09-15T04:05:09.764 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f574ffff640 1 -- 172.21.15.110:0/1144812383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750135930 msgr2=0x7f5750137d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:09.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f574ffff640 1 --2- 172.21.15.110:0/1144812383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750135930 0x7f5750137d10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f575006d930 tx=0x7f5748009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94be6a7def8ba68a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f574e7fc640 1 -- 172.21.15.110:0/1144812383 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748007840 con 0x7f5750135930 2024-09-15T04:05:09.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f574e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f574e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:09.765 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f574e7fc640 10 monclient: dump: 2024-09-15T04:05:09.765 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.765 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.765 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.765 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.765 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.765 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.765 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.766 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.766 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.766 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f574e7fc640 10 monclient: _finish_auth 0 2024-09-15T04:05:09.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f574e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:09.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f5755b5d640 5 monclient: authenticate success, global_id 55245 2024-09-15T04:05:09.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f5755b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:09.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f574e7fc640 1 -- 172.21.15.110:0/1144812383 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574802d050 con 0x7f5750135930 2024-09-15T04:05:09.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f574e7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:09.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f5755b5d640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:09.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f5755b5d640 4 set_mon_vals no callback set 2024-09-15T04:05:09.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f574e7fc640 1 -- 172.21.15.110:0/1144812383 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748007bd0 con 0x7f5750135930 2024-09-15T04:05:09.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f574e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.766 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f574e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.757+0000 7f574e7fc640 10 monclient: dump: 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.758+0000 7f5755b5d640 10 monclient: shutdown 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.758+0000 7f5755b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.758+0000 7f5755b5d640 1 -- 172.21.15.110:0/1144812383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750135930 msgr2=0x7f5750137d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.767 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.758+0000 7f5755b5d640 1 --2- 172.21.15.110:0/1144812383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750135930 0x7f5750137d10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f575006d930 tx=0x7f5748009780 comp rx=0 tx=0).stop 2024-09-15T04:05:09.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.758+0000 7f5755b5d640 1 -- 172.21.15.110:0/1144812383 shutdown_connections 2024-09-15T04:05:09.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.758+0000 7f5755b5d640 1 --2- 172.21.15.110:0/1144812383 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750135930 0x7f5750137d10 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.758+0000 7f5755b5d640 1 -- 172.21.15.110:0/1144812383 >> 172.21.15.110:0/1144812383 conn(0x7f575009e5e0 msgr2=0x7f575009e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:09.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.758+0000 7f5755b5d640 1 -- 172.21.15.110:0/1144812383 shutdown_connections 2024-09-15T04:05:09.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.758+0000 7f5755b5d640 1 -- 172.21.15.110:0/1144812383 wait complete. 2024-09-15T04:05:09.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.758+0000 7f5755b5d640 10 monclient: build_initial_monmap 2024-09-15T04:05:09.768 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.758+0000 7f5755b5d640 10 monclient: monmap: 2024-09-15T04:05:09.768 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:09.768 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:09.768 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:09.768 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:09.768 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:09.768 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.769 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:09.769 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:09.769 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:09.769 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 1 Processor -- start 2024-09-15T04:05:09.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 1 -- start start 2024-09-15T04:05:09.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 10 monclient: init 2024-09-15T04:05:09.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:09.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T04:05:09.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5750067130 0x7f575012feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f5750067130 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:09.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750067500 0x7f57500980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f5750067500 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:09.769 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5750135930 0x7f5750098630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f5750135930 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5750118a60 con 0x7f5750067500 2024-09-15T04:05:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575009a1b0 con 0x7f5750135930 2024-09-15T04:05:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575012f3a0 con 0x7f5750067130 2024-09-15T04:05:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f5755b5d640 10 monclient(hunting): authenticate will time out at 1133.403534s 2024-09-15T04:05:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750067500 0x7f57500980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5750067130 0x7f575012feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574effd640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750067500 0x7f57500980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54650/0 (socket says 172.21.15.110:54650) 2024-09-15T04:05:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5750067130 0x7f575012feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:37186/0 (socket says 172.21.15.110:37186) 2024-09-15T04:05:09.770 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574effd640 1 -- 172.21.15.110:0/4195164196 learned_addr learned my addr 172.21.15.110:0/4195164196 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574effd640 10 monclient(hunting): get_auth_request con 0x7f5750067500 auth_method 0 2024-09-15T04:05:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 10 monclient(hunting): get_auth_request con 0x7f5750067130 auth_method 0 2024-09-15T04:05:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.771 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574ffff640 1 --2- 172.21.15.110:0/4195164196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5750135930 0x7f5750098630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574ffff640 10 monclient(hunting): get_auth_request con 0x7f5750135930 auth_method 0 2024-09-15T04:05:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 10 monclient(hunting): handle_auth_done global_id 74198 payload 824 2024-09-15T04:05:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 1 -- 172.21.15.110:0/4195164196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5750135930 msgr2=0x7f5750098630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 1 --2- 172.21.15.110:0/4195164196 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5750135930 0x7f5750098630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 1 -- 172.21.15.110:0/4195164196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750067500 msgr2=0x7f57500980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 1 --2- 172.21.15.110:0/4195164196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750067500 0x7f57500980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 10 monclient: _finish_hunting 0 2024-09-15T04:05:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 1 monclient: found mon.noname-c 2024-09-15T04:05:09.772 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574f7fe640 1 -- 172.21.15.110:0/4195164196 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5748009a20 con 0x7f5750067130 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.759+0000 7f574effd640 1 --2- 172.21.15.110:0/4195164196 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5750067500 0x7f57500980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f574f7fe640 1 -- 172.21.15.110:0/4195164196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5750067130 msgr2=0x7f575012feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f574f7fe640 1 --2- 172.21.15.110:0/4195164196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5750067130 0x7f575012feb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5740009fd0 tx=0x7f5740009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f57377fe640 1 -- 172.21.15.110:0/4195164196 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574000f720 con 0x7f5750067130 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f57377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f57377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f57377fe640 10 monclient: dump: 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.773 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.774 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.774 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.774 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.774 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f57377fe640 10 monclient: _finish_auth 0 2024-09-15T04:05:09.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f57377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:09.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f57377fe640 1 -- 172.21.15.110:0/4195164196 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574000f880 con 0x7f5750067130 2024-09-15T04:05:09.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f5755b5d640 5 monclient: authenticate success, global_id 74198 2024-09-15T04:05:09.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f57377fe640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:09.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f5755b5d640 10 monclient: _renew_subs 2024-09-15T04:05:09.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f57377fe640 1 -- 172.21.15.110:0/4195164196 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574000fc40 con 0x7f5750067130 2024-09-15T04:05:09.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f5755b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f5755b5d640 1 -- 172.21.15.110:0/4195164196 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57501306f0 con 0x7f5750067130 2024-09-15T04:05:09.774 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f57377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f57377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f574dffb640 4 set_mon_vals no callback set 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f57377fe640 10 monclient: dump: 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f5755b5d640 10 monclient: _renew_subs 2024-09-15T04:05:09.775 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f5755b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.762+0000 7f5755b5d640 1 -- 172.21.15.110:0/4195164196 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5750099c80 con 0x7f5750067130 2024-09-15T04:05:09.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.763+0000 7f5755b5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:09.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.763+0000 7f5755b5d640 1 -- 172.21.15.110:0/4195164196 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5750070950 con 0x7f5750067130 2024-09-15T04:05:09.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.763+0000 7f57377fe640 1 -- 172.21.15.110:0/4195164196 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f5740011610 con 0x7f5750067130 2024-09-15T04:05:09.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.763+0000 7f57377fe640 1 --2- 172.21.15.110:0/4195164196 >> v2:172.21.15.177:6812/28201 conn(0x7f573c038700 0x7f573c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.763+0000 7f57377fe640 1 -- 172.21.15.110:0/4195164196 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f574004af60 con 0x7f5750067130 2024-09-15T04:05:09.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.763+0000 7f57377fe640 1 --2- 172.21.15.110:0/4195164196 >> v2:172.21.15.177:6800/28629 conn(0x7f573c03e1a0 0x7f573c0405b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.763+0000 7f57377fe640 1 -- 172.21.15.110:0/4195164196 --> v2:172.21.15.177:6800/28629 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f573c040c20 con 0x7f573c03e1a0 2024-09-15T04:05:09.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.763+0000 7f574effd640 1 --2- 172.21.15.110:0/4195164196 >> v2:172.21.15.177:6812/28201 conn(0x7f573c038700 0x7f573c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.763+0000 7f574effd640 10 monclient: get_auth_request con 0x7f573c038700 auth_method 0 2024-09-15T04:05:09.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.763+0000 7f574ffff640 1 --2- 172.21.15.110:0/4195164196 >> v2:172.21.15.177:6800/28629 conn(0x7f573c03e1a0 0x7f573c0405b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.776 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.763+0000 7f574ffff640 10 monclient: get_auth_request con 0x7f573c03e1a0 auth_method 0 2024-09-15T04:05:09.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.766+0000 7f574effd640 1 --2- 172.21.15.110:0/4195164196 >> v2:172.21.15.177:6812/28201 conn(0x7f573c038700 0x7f573c03abb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5744006fd0 tx=0x7f5744004960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.768+0000 7f574ffff640 1 --2- 172.21.15.110:0/4195164196 >> v2:172.21.15.177:6800/28629 conn(0x7f573c03e1a0 0x7f573c0405b0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.777 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.769+0000 7f57377fe640 1 -- 172.21.15.110:0/4195164196 <== osd.4 v2:172.21.15.177:6800/28629 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7f573c040c20 con 0x7f573c03e1a0 2024-09-15T04:05:09.780 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.780+0000 7f5755b5d640 1 -- 172.21.15.110:0/4195164196 --> v2:172.21.15.177:6800/28629 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f575009aa90 con 0x7f573c03e1a0 2024-09-15T04:05:09.781 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.780+0000 7f57377fe640 1 -- 172.21.15.110:0/4195164196 <== osd.4 v2:172.21.15.177:6800/28629 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f575009aa90 con 0x7f573c03e1a0 2024-09-15T04:05:09.781 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:05:09.781 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T04:05:09.781 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:05:09.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.781+0000 7f5755b5d640 1 -- 172.21.15.110:0/4195164196 >> v2:172.21.15.177:6800/28629 conn(0x7f573c03e1a0 msgr2=0x7f573c0405b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.781+0000 7f5755b5d640 1 --2- 172.21.15.110:0/4195164196 >> v2:172.21.15.177:6800/28629 conn(0x7f573c03e1a0 0x7f573c0405b0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.781+0000 7f5755b5d640 1 -- 172.21.15.110:0/4195164196 >> v2:172.21.15.177:6812/28201 conn(0x7f573c038700 msgr2=0x7f573c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.781+0000 7f5755b5d640 1 --2- 172.21.15.110:0/4195164196 >> v2:172.21.15.177:6812/28201 conn(0x7f573c038700 0x7f573c03abb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5744006fd0 tx=0x7f5744004960 comp rx=0 tx=0).stop 2024-09-15T04:05:09.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.781+0000 7f5755b5d640 10 monclient: shutdown 2024-09-15T04:05:09.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.781+0000 7f5755b5d640 20 monclient: shutdown canceling and discarding version request 1 2024-09-15T04:05:09.782 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.781+0000 7f5755b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:09.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.781+0000 7f5755b5d640 1 -- 172.21.15.110:0/4195164196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5750067130 msgr2=0x7f575012feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.781+0000 7f5755b5d640 1 --2- 172.21.15.110:0/4195164196 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f5750067130 0x7f575012feb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5740009fd0 tx=0x7f5740009060 comp rx=0 tx=0).stop 2024-09-15T04:05:09.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.781+0000 7f574ffff640 1 -- 172.21.15.110:0/4195164196 reap_dead start 2024-09-15T04:05:09.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.781+0000 7f5755b5d640 1 -- 172.21.15.110:0/4195164196 shutdown_connections 2024-09-15T04:05:09.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.781+0000 7f5755b5d640 1 -- 172.21.15.110:0/4195164196 >> 172.21.15.110:0/4195164196 conn(0x7f575009e5e0 msgr2=0x7f57501351e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:09.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.781+0000 7f5755b5d640 1 -- 172.21.15.110:0/4195164196 shutdown_connections 2024-09-15T04:05:09.783 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.781+0000 7f5755b5d640 1 -- 172.21.15.110:0/4195164196 wait complete. 2024-09-15T04:05:09.791 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-15T04:05:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.883+0000 7f6f4651d640 10 monclient: get_monmap_and_config 2024-09-15T04:05:09.884 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.884+0000 7f6f4651d640 10 monclient: build_initial_monmap 2024-09-15T04:05:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.884+0000 7f6f4651d640 10 monclient: monmap: 2024-09-15T04:05:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:09.885008+0000 2024-09-15T04:05:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:09.885008+0000 2024-09-15T04:05:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:09.885 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:09.885 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.887 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.884+0000 7f6f4651d640 1 Processor -- start 2024-09-15T04:05:09.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.884+0000 7f6f4651d640 1 -- start start 2024-09-15T04:05:09.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.884+0000 7f6f4651d640 10 monclient: init 2024-09-15T04:05:09.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f4651d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:09.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f4651d640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-15T04:05:09.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f4651d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f6f400a1cf0 0x7f6f4009e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f4651d640 10 monclient(hunting): picked mon.noname-c con 0x7f6f400a1cf0 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:09.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f4651d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f40131a80 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:09.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f4651d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6f40136940 0x7f6f40138d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f4651d640 10 monclient(hunting): picked mon.noname-a con 0x7f6f40136940 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:09.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f4651d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4009cb70 con 0x7f6f40131a80 2024-09-15T04:05:09.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f4651d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f4651d640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f400a20c0 con 0x7f6f40136940 2024-09-15T04:05:09.888 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f4651d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f4651d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4009c330 con 0x7f6f400a1cf0 2024-09-15T04:05:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f4651d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f4651d640 10 monclient(hunting): authenticate will time out at 1133.529307s 2024-09-15T04:05:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6f40136940 0x7f6f40138d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f3f7fe640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f6f40131a80 0x7f6f4009eba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51508/0 (socket says 172.21.15.110:51508) 2024-09-15T04:05:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f3f7fe640 1 -- 172.21.15.110:0/2653826615 learned_addr learned my addr 172.21.15.110:0/2653826615 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 10 monclient(hunting): get_auth_request con 0x7f6f40136940 auth_method 0 2024-09-15T04:05:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.889 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f3ffff640 1 --2- 172.21.15.110:0/2653826615 >> v2:172.21.15.177:3300/0 conn(0x7f6f400a1cf0 0x7f6f4009e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f3ffff640 10 monclient(hunting): get_auth_request con 0x7f6f400a1cf0 auth_method 0 2024-09-15T04:05:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 10 monclient(hunting): handle_auth_done global_id 55251 payload 274 2024-09-15T04:05:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 1 -- 172.21.15.110:0/2653826615 >> v2:172.21.15.177:3300/0 conn(0x7f6f400a1cf0 msgr2=0x7f6f4009e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 1 --2- 172.21.15.110:0/2653826615 >> v2:172.21.15.177:3300/0 conn(0x7f6f400a1cf0 0x7f6f4009e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 1 -- 172.21.15.110:0/2653826615 >> v1:172.21.15.110:6789/0 conn(0x7f6f40131a80 legacy=0x7f6f4009eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 10 monclient: _finish_hunting 0 2024-09-15T04:05:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 1 monclient: found mon.noname-a 2024-09-15T04:05:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:09.890 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:09.891 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.885+0000 7f6f44aba640 1 -- 172.21.15.110:0/2653826615 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f40131fb0 con 0x7f6f40136940 2024-09-15T04:05:09.896 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f44aba640 1 -- 172.21.15.110:0/2653826615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f40136940 msgr2=0x7f6f40138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:09.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f44aba640 1 --2- 172.21.15.110:0/2653826615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f40136940 0x7f6f40138d20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6f40063d70 tx=0x7f6f38009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37d080c5d468502 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f3effd640 1 -- 172.21.15.110:0/2653826615 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f38007760 con 0x7f6f40136940 2024-09-15T04:05:09.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f3effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:09.897 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f3effd640 10 monclient: dump: 2024-09-15T04:05:09.897 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.897 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.897 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.897 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.897 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.897 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.897 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.897 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.898 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.898 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f3effd640 10 monclient: _finish_auth 0 2024-09-15T04:05:09.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:09.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f4651d640 5 monclient: authenticate success, global_id 55251 2024-09-15T04:05:09.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f3effd640 1 -- 172.21.15.110:0/2653826615 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3802d050 con 0x7f6f40136940 2024-09-15T04:05:09.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f4651d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:09.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f3effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:09.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f4651d640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:09.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f3effd640 1 -- 172.21.15.110:0/2653826615 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f38007af0 con 0x7f6f40136940 2024-09-15T04:05:09.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f4651d640 4 set_mon_vals no callback set 2024-09-15T04:05:09.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f3effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:09.898 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.888+0000 7f6f3effd640 10 monclient: dump: 2024-09-15T04:05:09.899 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.899 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.899 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.899 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.899 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.899 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.899 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.899 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.899 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.899 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.889+0000 7f6f4651d640 10 monclient: shutdown 2024-09-15T04:05:09.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.889+0000 7f6f4651d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:09.899 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.889+0000 7f6f4651d640 1 -- 172.21.15.110:0/2653826615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f40136940 msgr2=0x7f6f40138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.889+0000 7f6f4651d640 1 --2- 172.21.15.110:0/2653826615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f40136940 0x7f6f40138d20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6f40063d70 tx=0x7f6f38009780 comp rx=0 tx=0).stop 2024-09-15T04:05:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.889+0000 7f6f4651d640 1 -- 172.21.15.110:0/2653826615 shutdown_connections 2024-09-15T04:05:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.889+0000 7f6f4651d640 1 --2- 172.21.15.110:0/2653826615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f40136940 0x7f6f40138d20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.889+0000 7f6f4651d640 1 --2- 172.21.15.110:0/2653826615 >> v2:172.21.15.177:3300/0 conn(0x7f6f400a1cf0 0x7f6f4009e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.889+0000 7f6f4651d640 1 -- 172.21.15.110:0/2653826615 >> 172.21.15.110:0/2653826615 conn(0x7f6f40062f70 msgr2=0x7f6f40063340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.889+0000 7f6f4651d640 1 -- 172.21.15.110:0/2653826615 shutdown_connections 2024-09-15T04:05:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.889+0000 7f6f4651d640 1 -- 172.21.15.110:0/2653826615 wait complete. 2024-09-15T04:05:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.889+0000 7f6f4651d640 10 monclient: build_initial_monmap 2024-09-15T04:05:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.889+0000 7f6f4651d640 10 monclient: monmap: 2024-09-15T04:05:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:09.900 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.889+0000 7f6f4651d640 1 Processor -- start 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 1 -- start start 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 10 monclient: init 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f400a1cf0 0x7f6f40140db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 10 monclient(hunting): picked mon.noname-c con 0x7f6f400a1cf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f40131a80 0x7f6f401412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.901 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 10 monclient(hunting): picked mon.noname-a con 0x7f6f40131a80 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f40136940 0x7f6f40141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 10 monclient(hunting): picked mon.noname-b con 0x7f6f40136940 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f400a11e0 con 0x7f6f40131a80 2024-09-15T04:05:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f400a5610 con 0x7f6f40136940 2024-09-15T04:05:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4009b080 con 0x7f6f400a1cf0 2024-09-15T04:05:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f4651d640 10 monclient(hunting): authenticate will time out at 1133.534565s 2024-09-15T04:05:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f40131a80 0x7f6f401412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f400a1cf0 0x7f6f40140db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f40131a80 0x7f6f401412f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54670/0 (socket says 172.21.15.110:54670) 2024-09-15T04:05:09.902 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3ffff640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f400a1cf0 0x7f6f40140db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:37198/0 (socket says 172.21.15.110:37198) 2024-09-15T04:05:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 1 -- 172.21.15.110:0/306552360 learned_addr learned my addr 172.21.15.110:0/306552360 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 10 monclient(hunting): get_auth_request con 0x7f6f40131a80 auth_method 0 2024-09-15T04:05:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3ffff640 10 monclient(hunting): get_auth_request con 0x7f6f400a1cf0 auth_method 0 2024-09-15T04:05:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f44aba640 1 --2- 172.21.15.110:0/306552360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f40136940 0x7f6f40141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.903 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f44aba640 10 monclient(hunting): get_auth_request con 0x7f6f40136940 auth_method 0 2024-09-15T04:05:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f44aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f44aba640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f44aba640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f44aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f44aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f44aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 10 monclient(hunting): handle_auth_done global_id 55254 payload 824 2024-09-15T04:05:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 1 -- 172.21.15.110:0/306552360 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f400a1cf0 msgr2=0x7f6f40140db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 1 --2- 172.21.15.110:0/306552360 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f400a1cf0 0x7f6f40140db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 1 -- 172.21.15.110:0/306552360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f40136940 msgr2=0x7f6f40141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:09.904 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 1 --2- 172.21.15.110:0/306552360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f40136940 0x7f6f40141830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 10 monclient: _finish_hunting 0 2024-09-15T04:05:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 1 monclient: found mon.noname-a 2024-09-15T04:05:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.890+0000 7f6f3f7fe640 1 -- 172.21.15.110:0/306552360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f38009a20 con 0x7f6f40131a80 2024-09-15T04:05:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f3f7fe640 1 -- 172.21.15.110:0/306552360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f40131a80 msgr2=0x7f6f401412f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f3f7fe640 1 --2- 172.21.15.110:0/306552360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f40131a80 0x7f6f401412f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400c5e0 tx=0x7f6f34004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f27fff640 1 -- 172.21.15.110:0/306552360 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f34005cb0 con 0x7f6f40131a80 2024-09-15T04:05:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f27fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f27fff640 10 monclient: dump: 2024-09-15T04:05:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.905 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f27fff640 10 monclient: _finish_auth 0 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f27fff640 1 -- 172.21.15.110:0/306552360 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f34005e30 con 0x7f6f40131a80 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f4651d640 5 monclient: authenticate success, global_id 55254 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f4651d640 10 monclient: _renew_subs 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f4651d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f4651d640 1 -- 172.21.15.110:0/306552360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4009b7d0 con 0x7f6f40131a80 2024-09-15T04:05:09.906 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f27fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f27fff640 1 -- 172.21.15.110:0/306552360 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f34007340 con 0x7f6f40131a80 2024-09-15T04:05:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f27fff640 10 monclient: dump: 2024-09-15T04:05:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:09.907 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:09.908 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f4651d640 10 monclient: _renew_subs 2024-09-15T04:05:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f4651d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f3e7fc640 4 set_mon_vals no callback set 2024-09-15T04:05:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.893+0000 7f6f4651d640 1 -- 172.21.15.110:0/306552360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4011b560 con 0x7f6f40131a80 2024-09-15T04:05:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.894+0000 7f6f27fff640 1 -- 172.21.15.110:0/306552360 <== mon.0 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f6f3401b400 con 0x7f6f40131a80 2024-09-15T04:05:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.894+0000 7f6f4651d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.894+0000 7f6f4651d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.894+0000 7f6f4651d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.110:3300/0 2024-09-15T04:05:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.894+0000 7f6f4651d640 1 -- 172.21.15.110:0/306552360 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4011c560 con 0x7f6f40131a80 2024-09-15T04:05:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.894+0000 7f6f27fff640 1 --2- 172.21.15.110:0/306552360 >> v2:172.21.15.177:6812/28201 conn(0x7f6f2c038700 0x7f6f2c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.894+0000 7f6f27fff640 1 -- 172.21.15.110:0/306552360 <== mon.0 v2:172.21.15.110:3300/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6f3404b1a0 con 0x7f6f40131a80 2024-09-15T04:05:09.908 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.894+0000 7f6f3ffff640 1 --2- 172.21.15.110:0/306552360 >> v2:172.21.15.177:6812/28201 conn(0x7f6f2c038700 0x7f6f2c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.894+0000 7f6f3ffff640 10 monclient: get_auth_request con 0x7f6f2c038700 auth_method 0 2024-09-15T04:05:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.897+0000 7f6f3ffff640 1 --2- 172.21.15.110:0/306552360 >> v2:172.21.15.177:6812/28201 conn(0x7f6f2c038700 0x7f6f2c03abb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000bd40 tx=0x7f6f30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.898+0000 7f6f27fff640 1 -- 172.21.15.110:0/306552360 <== mon.0 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f6f3401b6b0 con 0x7f6f40131a80 2024-09-15T04:05:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.898+0000 7f6f27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:09.909 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:09.898+0000 7f6f27fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:10.056 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.055+0000 7f6f4651d640 1 -- 172.21.15.110:0/306552360 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f4011c7a0 con 0x7f6f2c038700 2024-09-15T04:05:10.057 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.056+0000 7f6f27fff640 1 -- 172.21.15.110:0/306552360 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6f4011c7a0 con 0x7f6f2c038700 2024-09-15T04:05:10.057 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 4 to deep-scrub 2024-09-15T04:05:10.059 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.058+0000 7f6f4651d640 1 -- 172.21.15.110:0/306552360 >> v2:172.21.15.177:6812/28201 conn(0x7f6f2c038700 msgr2=0x7f6f2c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.058+0000 7f6f4651d640 1 --2- 172.21.15.110:0/306552360 >> v2:172.21.15.177:6812/28201 conn(0x7f6f2c038700 0x7f6f2c03abb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000bd40 tx=0x7f6f30008040 comp rx=0 tx=0).stop 2024-09-15T04:05:10.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.058+0000 7f6f4651d640 10 monclient: shutdown 2024-09-15T04:05:10.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.058+0000 7f6f4651d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:10.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.058+0000 7f6f4651d640 1 -- 172.21.15.110:0/306552360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f40131a80 msgr2=0x7f6f401412f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.058+0000 7f6f4651d640 1 --2- 172.21.15.110:0/306552360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f40131a80 0x7f6f401412f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400c5e0 tx=0x7f6f34004960 comp rx=0 tx=0).stop 2024-09-15T04:05:10.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.058+0000 7f6f4651d640 1 -- 172.21.15.110:0/306552360 shutdown_connections 2024-09-15T04:05:10.060 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.058+0000 7f6f4651d640 1 --2- 172.21.15.110:0/306552360 >> v2:172.21.15.177:6812/28201 conn(0x7f6f2c038700 0x7f6f2c03abb0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.058+0000 7f6f4651d640 1 --2- 172.21.15.110:0/306552360 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6f40136940 0x7f6f40141830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.058+0000 7f6f4651d640 1 --2- 172.21.15.110:0/306552360 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6f40131a80 0x7f6f401412f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.058+0000 7f6f4651d640 1 --2- 172.21.15.110:0/306552360 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f6f400a1cf0 0x7f6f40140db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.058+0000 7f6f4651d640 1 -- 172.21.15.110:0/306552360 >> 172.21.15.110:0/306552360 conn(0x7f6f40062f70 msgr2=0x7f6f400a2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:10.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.058+0000 7f6f4651d640 1 -- 172.21.15.110:0/306552360 shutdown_connections 2024-09-15T04:05:10.061 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.058+0000 7f6f4651d640 1 -- 172.21.15.110:0/306552360 wait complete. 2024-09-15T04:05:10.069 INFO:tasks.ceph:Scrubbing osd.5 2024-09-15T04:05:10.070 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-15T04:05:10.166 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.165+0000 7fcdf4c78640 10 monclient: get_monmap_and_config 2024-09-15T04:05:10.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.165+0000 7fcdf4c78640 10 monclient: build_initial_monmap 2024-09-15T04:05:10.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.165+0000 7fcdf4c78640 10 monclient: monmap: 2024-09-15T04:05:10.270 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:10.166664+0000 2024-09-15T04:05:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:10.166664+0000 2024-09-15T04:05:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:10.271 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 1 Processor -- start 2024-09-15T04:05:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 1 -- start start 2024-09-15T04:05:10.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 10 monclient: init 2024-09-15T04:05:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-15T04:05:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcdf0067130 0x7fcdf009dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 10 monclient(hunting): picked mon.noname-a con 0x7fcdf0067130 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcdf0067500 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcdf0134920 0x7fcdf0136d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 10 monclient(hunting): picked mon.noname-b con 0x7fcdf0134920 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:10.280 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf009c670 con 0x7fcdf0067500 2024-09-15T04:05:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf009cd70 con 0x7fcdf0067130 2024-09-15T04:05:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf009e260 con 0x7fcdf0134920 2024-09-15T04:05:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.166+0000 7fcdf4c78640 10 monclient(hunting): authenticate will time out at 1133.810967s 2024-09-15T04:05:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdee59c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcdf0067130 0x7fcdf009dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcdf0134920 0x7fcdf0136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdee59c640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcdf0067130 0x7fcdf009dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54672/0 (socket says 172.21.15.110:54672) 2024-09-15T04:05:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcdf0134920 0x7fcdf0136d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:37206/0 (socket says 172.21.15.110:37206) 2024-09-15T04:05:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdee59c640 1 -- 172.21.15.110:0/2161884132 learned_addr learned my addr 172.21.15.110:0/2161884132 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdee59c640 10 monclient(hunting): get_auth_request con 0x7fcdf0067130 auth_method 0 2024-09-15T04:05:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:10.281 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdee59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 10 monclient(hunting): get_auth_request con 0x7fcdf0134920 auth_method 0 2024-09-15T04:05:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 10 monclient(hunting): handle_auth_done global_id 74204 payload 274 2024-09-15T04:05:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 1 -- 172.21.15.110:0/2161884132 >> v2:172.21.15.110:3300/0 conn(0x7fcdf0067130 msgr2=0x7fcdf009dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:10.282 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 1 --2- 172.21.15.110:0/2161884132 >> v2:172.21.15.110:3300/0 conn(0x7fcdf0067130 0x7fcdf009dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 1 -- 172.21.15.110:0/2161884132 >> v1:172.21.15.177:6789/0 conn(0x7fcdf0067500 legacy=0x7fcdf0067900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 10 monclient: _finish_hunting 0 2024-09-15T04:05:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 1 monclient: found mon.noname-b 2024-09-15T04:05:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:05:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdeed9d640 1 -- 172.21.15.110:0/2161884132 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf009ca90 con 0x7fcdf0134920 2024-09-15T04:05:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.167+0000 7fcdee59c640 1 --2- 172.21.15.110:0/2161884132 >> v2:172.21.15.110:3300/0 conn(0x7fcdf0067130 0x7fcdf009dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdeed9d640 1 -- 172.21.15.110:0/2161884132 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcdf0134920 msgr2=0x7fcdf0136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdeed9d640 1 --2- 172.21.15.110:0/2161884132 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcdf0134920 0x7fcdf0136d00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcdf006d930 tx=0x7fcde8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2412d1758ea054c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:10.283 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcded59a640 1 -- 172.21.15.110:0/2161884132 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8007760 con 0x7fcdf0134920 2024-09-15T04:05:10.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcded59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:10.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcded59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:05:10.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcded59a640 10 monclient: dump: 2024-09-15T04:05:10.284 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:10.284 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.284 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.284 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.284 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:10.284 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.284 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:10.284 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:10.284 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:10.284 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.284 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcded59a640 10 monclient: _finish_auth 0 2024-09-15T04:05:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcded59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcded59a640 1 -- 172.21.15.110:0/2161884132 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde802d050 con 0x7fcdf0134920 2024-09-15T04:05:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdf4c78640 5 monclient: authenticate success, global_id 74204 2024-09-15T04:05:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdf4c78640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcded59a640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdf4c78640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdf4c78640 4 set_mon_vals no callback set 2024-09-15T04:05:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcded59a640 1 -- 172.21.15.110:0/2161884132 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8007af0 con 0x7fcdf0134920 2024-09-15T04:05:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcded59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:10.285 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcded59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcded59a640 10 monclient: dump: 2024-09-15T04:05:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:10.286 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.286 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdf4c78640 10 monclient: shutdown 2024-09-15T04:05:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdf4c78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdf4c78640 1 -- 172.21.15.110:0/2161884132 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcdf0134920 msgr2=0x7fcdf0136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdf4c78640 1 --2- 172.21.15.110:0/2161884132 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcdf0134920 0x7fcdf0136d00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcdf006d930 tx=0x7fcde8009780 comp rx=0 tx=0).stop 2024-09-15T04:05:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdf4c78640 1 -- 172.21.15.110:0/2161884132 shutdown_connections 2024-09-15T04:05:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdf4c78640 1 --2- 172.21.15.110:0/2161884132 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcdf0134920 0x7fcdf0136d00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdf4c78640 1 --2- 172.21.15.110:0/2161884132 >> v2:172.21.15.110:3300/0 conn(0x7fcdf0067130 0x7fcdf009dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdf4c78640 1 -- 172.21.15.110:0/2161884132 >> 172.21.15.110:0/2161884132 conn(0x7fcdf009e5e0 msgr2=0x7fcdf009e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdf4c78640 1 -- 172.21.15.110:0/2161884132 shutdown_connections 2024-09-15T04:05:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.170+0000 7fcdf4c78640 1 -- 172.21.15.110:0/2161884132 wait complete. 2024-09-15T04:05:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.171+0000 7fcdf4c78640 10 monclient: build_initial_monmap 2024-09-15T04:05:10.287 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.171+0000 7fcdf4c78640 10 monclient: monmap: 2024-09-15T04:05:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:10.288 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.171+0000 7fcdf4c78640 1 Processor -- start 2024-09-15T04:05:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.171+0000 7fcdf4c78640 1 -- start start 2024-09-15T04:05:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.171+0000 7fcdf4c78640 10 monclient: init 2024-09-15T04:05:10.288 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T04:05:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf0067130 0x7fcdf0141300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 10 monclient(hunting): picked mon.noname-b con 0x7fcdf0067130 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcdf0067500 0x7fcdf012fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 10 monclient(hunting): picked mon.noname-c con 0x7fcdf0067500 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdf0134920 0x7fcdf013ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 10 monclient(hunting): picked mon.noname-a con 0x7fcdf0134920 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf006cf40 con 0x7fcdf0134920 2024-09-15T04:05:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf01306f0 con 0x7fcdf0067130 2024-09-15T04:05:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf00a15b0 con 0x7fcdf0067500 2024-09-15T04:05:10.289 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdf4c78640 10 monclient(hunting): authenticate will time out at 1133.816138s 2024-09-15T04:05:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdeed9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdf0134920 0x7fcdf013ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcdf0067500 0x7fcdf012fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdeed9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdf0134920 0x7fcdf013ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54682/0 (socket says 172.21.15.110:54682) 2024-09-15T04:05:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 10 monclient(hunting): get_auth_request con 0x7fcdf0067500 auth_method 0 2024-09-15T04:05:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdeed9d640 1 -- 172.21.15.110:0/608109409 learned_addr learned my addr 172.21.15.110:0/608109409 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdeed9d640 10 monclient(hunting): get_auth_request con 0x7fcdf0134920 auth_method 0 2024-09-15T04:05:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdeed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdeed9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdeed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:10.290 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdeed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdeed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdeed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:10.291 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdee59c640 1 --2- 172.21.15.110:0/608109409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf0067130 0x7fcdf0141300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdee59c640 10 monclient(hunting): get_auth_request con 0x7fcdf0067130 auth_method 0 2024-09-15T04:05:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdee59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 10 monclient(hunting): handle_auth_done global_id 74207 payload 824 2024-09-15T04:05:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 1 -- 172.21.15.110:0/608109409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf0067130 msgr2=0x7fcdf0141300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 1 --2- 172.21.15.110:0/608109409 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcdf0067130 0x7fcdf0141300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 1 -- 172.21.15.110:0/608109409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdf0134920 msgr2=0x7fcdf013ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 1 --2- 172.21.15.110:0/608109409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcdf0134920 0x7fcdf013ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 10 monclient: _finish_hunting 0 2024-09-15T04:05:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 1 monclient: found mon.noname-c 2024-09-15T04:05:10.292 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:10.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:10.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.172+0000 7fcdedd9b640 1 -- 172.21.15.110:0/608109409 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde0009450 con 0x7fcdf0067500 2024-09-15T04:05:10.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdedd9b640 1 -- 172.21.15.110:0/608109409 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcdf0067500 msgr2=0x7fcdf012fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:10.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdedd9b640 1 --2- 172.21.15.110:0/608109409 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcdf0067500 0x7fcdf012fba0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcde400c510 tx=0x7fcde4007e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:10.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdde7fc640 1 -- 172.21.15.110:0/608109409 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde40117a0 con 0x7fcdf0067500 2024-09-15T04:05:10.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:10.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdde7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:10.293 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdde7fc640 10 monclient: dump: 2024-09-15T04:05:10.293 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:10.293 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.293 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.294 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.294 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:10.294 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.294 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:10.294 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:10.294 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:10.294 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdde7fc640 10 monclient: _finish_auth 0 2024-09-15T04:05:10.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:10.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdde7fc640 1 -- 172.21.15.110:0/608109409 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde4011900 con 0x7fcdf0067500 2024-09-15T04:05:10.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdf4c78640 5 monclient: authenticate success, global_id 74207 2024-09-15T04:05:10.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdf4c78640 10 monclient: _renew_subs 2024-09-15T04:05:10.294 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdf4c78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:10.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdf4c78640 1 -- 172.21.15.110:0/608109409 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde8009a20 con 0x7fcdf0067500 2024-09-15T04:05:10.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdde7fc640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:10.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdde7fc640 1 -- 172.21.15.110:0/608109409 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde4011ca0 con 0x7fcdf0067500 2024-09-15T04:05:10.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:10.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdde7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:10.295 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdde7fc640 10 monclient: dump: 2024-09-15T04:05:10.295 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:10.295 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.295 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.295 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.295 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:10.295 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:10.296 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdecd99640 4 set_mon_vals no callback set 2024-09-15T04:05:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdf4c78640 10 monclient: _renew_subs 2024-09-15T04:05:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdf4c78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.174+0000 7fcdf4c78640 1 -- 172.21.15.110:0/608109409 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf01300e0 con 0x7fcdf0067500 2024-09-15T04:05:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.175+0000 7fcdf4c78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.175+0000 7fcdf4c78640 1 -- 172.21.15.110:0/608109409 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcdf0141e70 con 0x7fcdf0067500 2024-09-15T04:05:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.175+0000 7fcdde7fc640 1 -- 172.21.15.110:0/608109409 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7fcde4012630 con 0x7fcdf0067500 2024-09-15T04:05:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.175+0000 7fcdde7fc640 1 --2- 172.21.15.110:0/608109409 >> v2:172.21.15.177:6812/28201 conn(0x7fcdd0038730 0x7fcdd003abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.296 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.175+0000 7fcdde7fc640 1 -- 172.21.15.110:0/608109409 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fcde404b300 con 0x7fcdf0067500 2024-09-15T04:05:10.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.176+0000 7fcdde7fc640 1 --2- 172.21.15.110:0/608109409 >> v2:172.21.15.177:6808/28630 conn(0x7fcdd003e250 0x7fcdd0040660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.176+0000 7fcdde7fc640 1 -- 172.21.15.110:0/608109409 --> v2:172.21.15.177:6808/28630 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcdd0040cd0 con 0x7fcdd003e250 2024-09-15T04:05:10.297 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.176+0000 7fcdde7fc640 1 -- 172.21.15.110:0/608109409 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_get_version_reply(handle=1 version=315) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcde404d820 con 0x7fcdf0067500 2024-09-15T04:05:10.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.176+0000 7fcdde7fc640 10 monclient: handle_get_version_reply finishing 1 version 315 2024-09-15T04:05:10.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.176+0000 7fcdee59c640 1 --2- 172.21.15.110:0/608109409 >> v2:172.21.15.177:6812/28201 conn(0x7fcdd0038730 0x7fcdd003abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.176+0000 7fcdee59c640 10 monclient: get_auth_request con 0x7fcdd0038730 auth_method 0 2024-09-15T04:05:10.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.176+0000 7fcdeed9d640 1 --2- 172.21.15.110:0/608109409 >> v2:172.21.15.177:6808/28630 conn(0x7fcdd003e250 0x7fcdd0040660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.176+0000 7fcdeed9d640 10 monclient: get_auth_request con 0x7fcdd003e250 auth_method 0 2024-09-15T04:05:10.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.178+0000 7fcdee59c640 1 --2- 172.21.15.110:0/608109409 >> v2:172.21.15.177:6812/28201 conn(0x7fcdd0038730 0x7fcdd003abe0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009420 tx=0x7fcde0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:10.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.180+0000 7fcdeed9d640 1 --2- 172.21.15.110:0/608109409 >> v2:172.21.15.177:6808/28630 conn(0x7fcdd003e250 0x7fcdd0040660 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:10.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.181+0000 7fcdde7fc640 1 -- 172.21.15.110:0/608109409 <== osd.5 v2:172.21.15.177:6808/28630 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+22488 (crc 0 0 0) 0x7fcdd0040cd0 con 0x7fcdd003e250 2024-09-15T04:05:10.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.191+0000 7fcdf4c78640 1 -- 172.21.15.110:0/608109409 --> v2:172.21.15.177:6808/28630 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fcdf0130330 con 0x7fcdd003e250 2024-09-15T04:05:10.298 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.192+0000 7fcdde7fc640 1 -- 172.21.15.110:0/608109409 <== osd.5 v2:172.21.15.177:6808/28630 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fcdf0130330 con 0x7fcdd003e250 2024-09-15T04:05:10.299 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-09-15T04:05:10.299 INFO:teuthology.orchestra.run.smithi110.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T04:05:10.299 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-09-15T04:05:10.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.192+0000 7fcdf4c78640 1 -- 172.21.15.110:0/608109409 >> v2:172.21.15.177:6808/28630 conn(0x7fcdd003e250 msgr2=0x7fcdd0040660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.302 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.192+0000 7fcdf4c78640 1 --2- 172.21.15.110:0/608109409 >> v2:172.21.15.177:6808/28630 conn(0x7fcdd003e250 0x7fcdd0040660 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.192+0000 7fcdf4c78640 1 -- 172.21.15.110:0/608109409 >> v2:172.21.15.177:6812/28201 conn(0x7fcdd0038730 msgr2=0x7fcdd003abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.192+0000 7fcdf4c78640 1 --2- 172.21.15.110:0/608109409 >> v2:172.21.15.177:6812/28201 conn(0x7fcdd0038730 0x7fcdd003abe0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcde0009420 tx=0x7fcde0008040 comp rx=0 tx=0).stop 2024-09-15T04:05:10.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.192+0000 7fcdf4c78640 10 monclient: shutdown 2024-09-15T04:05:10.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.192+0000 7fcdf4c78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:10.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.192+0000 7fcdf4c78640 1 -- 172.21.15.110:0/608109409 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcdf0067500 msgr2=0x7fcdf012fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.303 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.192+0000 7fcdf4c78640 1 --2- 172.21.15.110:0/608109409 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7fcdf0067500 0x7fcdf012fba0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcde400c510 tx=0x7fcde4007e10 comp rx=0 tx=0).stop 2024-09-15T04:05:10.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.192+0000 7fcdeed9d640 1 -- 172.21.15.110:0/608109409 reap_dead start 2024-09-15T04:05:10.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.192+0000 7fcdf4c78640 1 -- 172.21.15.110:0/608109409 shutdown_connections 2024-09-15T04:05:10.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.192+0000 7fcdf4c78640 1 -- 172.21.15.110:0/608109409 >> 172.21.15.110:0/608109409 conn(0x7fcdf009e5e0 msgr2=0x7fcdf00a04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:10.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.192+0000 7fcdf4c78640 1 -- 172.21.15.110:0/608109409 shutdown_connections 2024-09-15T04:05:10.304 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.192+0000 7fcdf4c78640 1 -- 172.21.15.110:0/608109409 wait complete. 2024-09-15T04:05:10.304 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-15T04:05:10.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.398+0000 7f34ec846640 10 monclient: get_monmap_and_config 2024-09-15T04:05:10.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.398+0000 7f34ec846640 10 monclient: build_initial_monmap 2024-09-15T04:05:10.399 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.398+0000 7f34ec846640 10 monclient: monmap: 2024-09-15T04:05:10.399 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:10.399 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:10.399641+0000 2024-09-15T04:05:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:10.399641+0000 2024-09-15T04:05:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:10.400 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:10.400 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 1 Processor -- start 2024-09-15T04:05:10.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 1 -- start start 2024-09-15T04:05:10.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 10 monclient: init 2024-09-15T04:05:10.404 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-15T04:05:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34e80a1cf0 addr v1:172.21.15.177:6789/0 2024-09-15T04:05:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8131a80 0x7f34e809ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 10 monclient(hunting): picked mon.noname-b con 0x7f34e8131a80 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f34e8136940 0x7f34e8138d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 10 monclient(hunting): picked mon.noname-a con 0x7f34e8136940 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 1 -- --> v1:172.21.15.177:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e809cb70 con 0x7f34e80a1cf0 2024-09-15T04:05:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e80a20c0 con 0x7f34e8136940 2024-09-15T04:05:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e809c330 con 0x7f34e8131a80 2024-09-15T04:05:10.405 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:10.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ec846640 10 monclient(hunting): authenticate will time out at 1134.043747s 2024-09-15T04:05:10.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34e6d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f34e8136940 0x7f34e8138d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ddd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8131a80 0x7f34e809ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ddd9b640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8131a80 0x7f34e809ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:37224/0 (socket says 172.21.15.110:37224) 2024-09-15T04:05:10.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34e6d9d640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f34e8136940 0x7f34e8138d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54686/0 (socket says 172.21.15.110:54686) 2024-09-15T04:05:10.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ddd9b640 1 -- 172.21.15.110:0/1430737725 learned_addr learned my addr 172.21.15.110:0/1430737725 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:10.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ddd9b640 10 monclient(hunting): get_auth_request con 0x7f34e8131a80 auth_method 0 2024-09-15T04:05:10.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:10.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ddd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:10.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:10.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34e6d9d640 10 monclient(hunting): get_auth_request con 0x7f34e8136940 auth_method 0 2024-09-15T04:05:10.406 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:10.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:10.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:10.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:10.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.399+0000 7f34ddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:10.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.400+0000 7f34ddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:10.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.400+0000 7f34e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:10.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.400+0000 7f34e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:10.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.400+0000 7f34e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:10.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.400+0000 7f34e6d9d640 10 monclient(hunting): handle_auth_done global_id 55263 payload 274 2024-09-15T04:05:10.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.400+0000 7f34e6d9d640 1 -- 172.21.15.110:0/1430737725 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8131a80 msgr2=0x7f34e809ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:10.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.400+0000 7f34e6d9d640 1 --2- 172.21.15.110:0/1430737725 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8131a80 0x7f34e809ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.400+0000 7f34e6d9d640 1 -- 172.21.15.110:0/1430737725 >> v1:172.21.15.177:6789/0 conn(0x7f34e80a1cf0 legacy=0x7f34e809e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:10.407 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.400+0000 7f34e6d9d640 10 monclient: _finish_hunting 0 2024-09-15T04:05:10.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.400+0000 7f34e6d9d640 1 monclient: found mon.noname-a 2024-09-15T04:05:10.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.400+0000 7f34e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:10.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.400+0000 7f34e6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:10.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.400+0000 7f34e6d9d640 1 -- 172.21.15.110:0/1430737725 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e8131fb0 con 0x7f34e8136940 2024-09-15T04:05:10.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.400+0000 7f34ddd9b640 1 --2- 172.21.15.110:0/1430737725 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8131a80 0x7f34e809ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:10.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.402+0000 7f34e6d9d640 1 -- 172.21.15.110:0/1430737725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34e8136940 msgr2=0x7f34e8138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:10.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.402+0000 7f34e6d9d640 1 --2- 172.21.15.110:0/1430737725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34e8136940 0x7f34e8138d20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f34e81321a0 tx=0x7f34e0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa39f9bd7f91fc15 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:10.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34e5d9b640 1 -- 172.21.15.110:0/1430737725 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0007760 con 0x7f34e8136940 2024-09-15T04:05:10.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:10.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34e5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:10.408 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34e5d9b640 10 monclient: dump: 2024-09-15T04:05:10.408 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:10.408 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.409 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:10.413 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34e5d9b640 10 monclient: _finish_auth 0 2024-09-15T04:05:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34e5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34ec846640 5 monclient: authenticate success, global_id 55263 2024-09-15T04:05:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34e5d9b640 1 -- 172.21.15.110:0/1430737725 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e002d050 con 0x7f34e8136940 2024-09-15T04:05:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34e5d9b640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34e5d9b640 1 -- 172.21.15.110:0/1430737725 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0007af0 con 0x7f34e8136940 2024-09-15T04:05:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34ec846640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34e5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:10.413 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34ec846640 4 set_mon_vals no callback set 2024-09-15T04:05:10.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34e5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:10.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34e5d9b640 10 monclient: dump: 2024-09-15T04:05:10.414 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:10.414 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.414 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.414 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.414 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:10.414 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.414 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:10.414 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:10.414 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:10.414 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34ec846640 10 monclient: shutdown 2024-09-15T04:05:10.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34ec846640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:10.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34ec846640 1 -- 172.21.15.110:0/1430737725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34e8136940 msgr2=0x7f34e8138d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34ec846640 1 --2- 172.21.15.110:0/1430737725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34e8136940 0x7f34e8138d20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f34e81321a0 tx=0x7f34e0009780 comp rx=0 tx=0).stop 2024-09-15T04:05:10.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34ec846640 1 -- 172.21.15.110:0/1430737725 shutdown_connections 2024-09-15T04:05:10.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34ec846640 1 --2- 172.21.15.110:0/1430737725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34e8136940 0x7f34e8138d20 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34ec846640 1 --2- 172.21.15.110:0/1430737725 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8131a80 0x7f34e809ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34ec846640 1 -- 172.21.15.110:0/1430737725 >> 172.21.15.110:0/1430737725 conn(0x7f34e8062f70 msgr2=0x7f34e8063340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:10.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34ec846640 1 -- 172.21.15.110:0/1430737725 shutdown_connections 2024-09-15T04:05:10.415 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.403+0000 7f34ec846640 1 -- 172.21.15.110:0/1430737725 wait complete. 2024-09-15T04:05:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.404+0000 7f34ec846640 10 monclient: build_initial_monmap 2024-09-15T04:05:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.404+0000 7f34ec846640 10 monclient: monmap: 2024-09-15T04:05:10.420 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:10.421 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.404+0000 7f34ec846640 1 Processor -- start 2024-09-15T04:05:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.404+0000 7f34ec846640 1 -- start start 2024-09-15T04:05:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.404+0000 7f34ec846640 10 monclient: init 2024-09-15T04:05:10.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T04:05:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34e80a1cf0 0x7f34e8135100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 10 monclient(hunting): picked mon.noname-a con 0x7f34e80a1cf0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34e8131a80 0x7f34e812fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 10 monclient(hunting): picked mon.noname-b con 0x7f34e8131a80 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8136940 0x7f34e8130380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 10 monclient(hunting): picked mon.noname-c con 0x7f34e8136940 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8134720 con 0x7f34e80a1cf0 2024-09-15T04:05:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8132990 con 0x7f34e8131a80 2024-09-15T04:05:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8134440 con 0x7f34e8136940 2024-09-15T04:05:10.422 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ec846640 10 monclient(hunting): authenticate will time out at 1134.049336s 2024-09-15T04:05:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8136940 0x7f34e8130380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e659c640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34e80a1cf0 0x7f34e8135100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8136940 0x7f34e8130380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:37228/0 (socket says 172.21.15.110:37228) 2024-09-15T04:05:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 1 -- 172.21.15.110:0/2222813389 learned_addr learned my addr 172.21.15.110:0/2222813389 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 10 monclient(hunting): get_auth_request con 0x7f34e8136940 auth_method 0 2024-09-15T04:05:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ddd9b640 1 --2- 172.21.15.110:0/2222813389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34e8131a80 0x7f34e812fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e659c640 10 monclient(hunting): get_auth_request con 0x7f34e80a1cf0 auth_method 0 2024-09-15T04:05:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e659c640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:10.423 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ddd9b640 10 monclient(hunting): get_auth_request con 0x7f34e8131a80 auth_method 0 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ddd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34ddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 10 monclient(hunting): handle_auth_done global_id 74213 payload 824 2024-09-15T04:05:10.424 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 1 -- 172.21.15.110:0/2222813389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34e8131a80 msgr2=0x7f34e812fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 1 --2- 172.21.15.110:0/2222813389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34e8131a80 0x7f34e812fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 1 -- 172.21.15.110:0/2222813389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34e80a1cf0 msgr2=0x7f34e8135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 1 --2- 172.21.15.110:0/2222813389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34e80a1cf0 0x7f34e8135100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 10 monclient: _finish_hunting 0 2024-09-15T04:05:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 1 monclient: found mon.noname-c 2024-09-15T04:05:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e6d9d640 1 -- 172.21.15.110:0/2222813389 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d0009450 con 0x7f34e8136940 2024-09-15T04:05:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.405+0000 7f34e659c640 1 --2- 172.21.15.110:0/2222813389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34e80a1cf0 0x7f34e8135100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34e6d9d640 1 -- 172.21.15.110:0/2222813389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8136940 msgr2=0x7f34e8130380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34e6d9d640 1 --2- 172.21.15.110:0/2222813389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8136940 0x7f34e8130380 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f34e00099f0 tx=0x7f34e0037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34deffd640 1 -- 172.21.15.110:0/2222813389 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0031b90 con 0x7f34e8136940 2024-09-15T04:05:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:10.425 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34deffd640 10 monclient: dump: 2024-09-15T04:05:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:10.426 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34deffd640 10 monclient: _finish_auth 0 2024-09-15T04:05:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:10.426 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34deffd640 1 -- 172.21.15.110:0/2222813389 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e0040070 con 0x7f34e8136940 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34ec846640 5 monclient: authenticate success, global_id 74213 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34ec846640 10 monclient: _renew_subs 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34ec846640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34ec846640 1 -- 172.21.15.110:0/2222813389 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e0009a20 con 0x7f34e8136940 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34deffd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34deffd640 1 -- 172.21.15.110:0/2222813389 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0045480 con 0x7f34e8136940 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34deffd640 10 monclient: dump: 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:10.427 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:10.428 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34e559a640 4 set_mon_vals no callback set 2024-09-15T04:05:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34ec846640 10 monclient: _renew_subs 2024-09-15T04:05:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34ec846640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34ec846640 1 -- 172.21.15.110:0/2222813389 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e8136520 con 0x7f34e8136940 2024-09-15T04:05:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.408+0000 7f34deffd640 1 -- 172.21.15.110:0/2222813389 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f34e0042030 con 0x7f34e8136940 2024-09-15T04:05:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.409+0000 7f34deffd640 1 --2- 172.21.15.110:0/2222813389 >> v2:172.21.15.177:6812/28201 conn(0x7f34c8038700 0x7f34c803abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.409+0000 7f34deffd640 1 -- 172.21.15.110:0/2222813389 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f34e006cca0 con 0x7f34e8136940 2024-09-15T04:05:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.409+0000 7f34ec846640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:10.428 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.409+0000 7f34ec846640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:10.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.409+0000 7f34ec846640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:10.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.409+0000 7f34e659c640 1 --2- 172.21.15.110:0/2222813389 >> v2:172.21.15.177:6812/28201 conn(0x7f34c8038700 0x7f34c803abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.409+0000 7f34ec846640 1 -- 172.21.15.110:0/2222813389 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34e80647e0 con 0x7f34e8136940 2024-09-15T04:05:10.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.409+0000 7f34e659c640 10 monclient: get_auth_request con 0x7f34c8038700 auth_method 0 2024-09-15T04:05:10.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.411+0000 7f34e659c640 1 --2- 172.21.15.110:0/2222813389 >> v2:172.21.15.177:6812/28201 conn(0x7f34c8038700 0x7f34c803abb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009e40 tx=0x7f34d8006f50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:10.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.413+0000 7f34deffd640 1 -- 172.21.15.110:0/2222813389 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f34e004a080 con 0x7f34e8136940 2024-09-15T04:05:10.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.413+0000 7f34deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:10.429 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.413+0000 7f34deffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:10.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.567+0000 7f34ec846640 1 -- 172.21.15.110:0/2222813389 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f34e8132e20 con 0x7f34c8038700 2024-09-15T04:05:10.568 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.568+0000 7f34deffd640 1 -- 172.21.15.110:0/2222813389 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f34e8132e20 con 0x7f34c8038700 2024-09-15T04:05:10.569 INFO:teuthology.orchestra.run.smithi110.stderr:instructed osd(s) 5 to deep-scrub 2024-09-15T04:05:10.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.570+0000 7f34ec846640 1 -- 172.21.15.110:0/2222813389 >> v2:172.21.15.177:6812/28201 conn(0x7f34c8038700 msgr2=0x7f34c803abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.570+0000 7f34ec846640 1 --2- 172.21.15.110:0/2222813389 >> v2:172.21.15.177:6812/28201 conn(0x7f34c8038700 0x7f34c803abb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f34d8009e40 tx=0x7f34d8006f50 comp rx=0 tx=0).stop 2024-09-15T04:05:10.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.570+0000 7f34ec846640 10 monclient: shutdown 2024-09-15T04:05:10.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.570+0000 7f34ec846640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:10.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.570+0000 7f34ec846640 1 -- 172.21.15.110:0/2222813389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8136940 msgr2=0x7f34e8130380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.570+0000 7f34ec846640 1 --2- 172.21.15.110:0/2222813389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8136940 0x7f34e8130380 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f34e00099f0 tx=0x7f34e0037000 comp rx=0 tx=0).stop 2024-09-15T04:05:10.571 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.570+0000 7f34ec846640 1 -- 172.21.15.110:0/2222813389 shutdown_connections 2024-09-15T04:05:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.570+0000 7f34ec846640 1 --2- 172.21.15.110:0/2222813389 >> v2:172.21.15.177:6812/28201 conn(0x7f34c8038700 0x7f34c803abb0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.570+0000 7f34ec846640 1 --2- 172.21.15.110:0/2222813389 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f34e8136940 0x7f34e8130380 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.570+0000 7f34ec846640 1 --2- 172.21.15.110:0/2222813389 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f34e8131a80 0x7f34e812fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.570+0000 7f34ec846640 1 --2- 172.21.15.110:0/2222813389 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34e80a1cf0 0x7f34e8135100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.570+0000 7f34ec846640 1 -- 172.21.15.110:0/2222813389 >> 172.21.15.110:0/2222813389 conn(0x7f34e8062f70 msgr2=0x7f34e812c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.570+0000 7f34ec846640 1 -- 172.21.15.110:0/2222813389 shutdown_connections 2024-09-15T04:05:10.572 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.570+0000 7f34ec846640 1 -- 172.21.15.110:0/2222813389 wait complete. 2024-09-15T04:05:10.581 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T04:05:10.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.672+0000 7ff4cbb9d640 10 monclient: get_monmap_and_config 2024-09-15T04:05:10.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.672+0000 7ff4cbb9d640 10 monclient: build_initial_monmap 2024-09-15T04:05:10.673 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.672+0000 7ff4cbb9d640 10 monclient: monmap: 2024-09-15T04:05:10.674 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:10.674 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.674 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:10.673926+0000 2024-09-15T04:05:10.674 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:10.673926+0000 2024-09-15T04:05:10.674 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:10.674 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.674 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:10.674 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:10.674 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:10.674 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:10.674 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:10.674 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.676 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.673+0000 7ff4cbb9d640 1 Processor -- start 2024-09-15T04:05:10.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.673+0000 7ff4cbb9d640 1 -- start start 2024-09-15T04:05:10.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.673+0000 7ff4cbb9d640 10 monclient: init 2024-09-15T04:05:10.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4cbb9d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:10.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4cbb9d640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-15T04:05:10.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4cbb9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40a1bf0 0x7ff4c4131810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4cbb9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4c40a1bf0 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:10.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4cbb9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff4c409e780 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:10.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4cbb9d640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7ff4c4136a40 0x7ff4c4138e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4cbb9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4c4136a40 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:10.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4cbb9d640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c409c9b0 con 0x7ff4c409e780 2024-09-15T04:05:10.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4cbb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4cbb9d640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c40a0c20 con 0x7ff4c4136a40 2024-09-15T04:05:10.677 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4cbb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4cbb9d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c409cc90 con 0x7ff4c40a1bf0 2024-09-15T04:05:10.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4cbb9d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:10.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4cbb9d640 10 monclient(hunting): authenticate will time out at 1134.318218s 2024-09-15T04:05:10.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40a1bf0 0x7ff4c4131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9138640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff4c409e780 0x7ff4c409eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:51510/0 (socket says 172.21.15.110:51510) 2024-09-15T04:05:10.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9138640 1 -- 172.21.15.110:0/625487616 learned_addr learned my addr 172.21.15.110:0/625487616 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:10.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 10 monclient(hunting): get_auth_request con 0x7ff4c40a1bf0 auth_method 0 2024-09-15T04:05:10.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:10.678 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:10.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:10.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:10.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:10.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:10.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 10 monclient(hunting): handle_auth_done global_id 74216 payload 274 2024-09-15T04:05:10.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 1 -- 172.21.15.110:0/625487616 >> v2:172.21.15.177:3300/0 conn(0x7ff4c4136a40 msgr2=0x7ff4c4138e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:10.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 1 --2- 172.21.15.110:0/625487616 >> v2:172.21.15.177:3300/0 conn(0x7ff4c4136a40 0x7ff4c4138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.679 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 1 -- 172.21.15.110:0/625487616 >> v1:172.21.15.110:6789/0 conn(0x7ff4c409e780 legacy=0x7ff4c409eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:10.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 10 monclient: _finish_hunting 0 2024-09-15T04:05:10.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 1 monclient: found mon.noname-b 2024-09-15T04:05:10.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:10.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:05:10.680 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.674+0000 7ff4c9939640 1 -- 172.21.15.110:0/625487616 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c40a1fc0 con 0x7ff4c40a1bf0 2024-09-15T04:05:10.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4c9939640 1 -- 172.21.15.110:0/625487616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40a1bf0 msgr2=0x7ff4c4131810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:10.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4c9939640 1 --2- 172.21.15.110:0/625487616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40a1bf0 0x7ff4c4131810 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff4c409d670 tx=0x7ff4c00090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=989106afd05a66ea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:10.686 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4c8937640 1 -- 172.21.15.110:0/625487616 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0032440 con 0x7ff4c40a1bf0 2024-09-15T04:05:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4c8937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4c8937640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:05:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4c8937640 10 monclient: dump: 2024-09-15T04:05:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:10.687 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.687 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4c8937640 10 monclient: _finish_auth 0 2024-09-15T04:05:10.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4c8937640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:10.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4c8937640 1 -- 172.21.15.110:0/625487616 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c00325a0 con 0x7ff4c40a1bf0 2024-09-15T04:05:10.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4cbb9d640 5 monclient: authenticate success, global_id 74216 2024-09-15T04:05:10.688 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4cbb9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:10.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4c8937640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:10.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4cbb9d640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:10.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4cbb9d640 4 set_mon_vals no callback set 2024-09-15T04:05:10.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4c8937640 1 -- 172.21.15.110:0/625487616 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0032940 con 0x7ff4c40a1bf0 2024-09-15T04:05:10.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4c8937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:10.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4c8937640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:10.689 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.677+0000 7ff4c8937640 10 monclient: dump: 2024-09-15T04:05:10.689 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:10.689 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.689 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:10.690 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.690 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.678+0000 7ff4cbb9d640 10 monclient: shutdown 2024-09-15T04:05:10.691 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.678+0000 7ff4cbb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.678+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/625487616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40a1bf0 msgr2=0x7ff4c4131810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.678+0000 7ff4cbb9d640 1 --2- 172.21.15.110:0/625487616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40a1bf0 0x7ff4c4131810 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff4c409d670 tx=0x7ff4c00090f0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.678+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/625487616 shutdown_connections 2024-09-15T04:05:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.678+0000 7ff4cbb9d640 1 --2- 172.21.15.110:0/625487616 >> v2:172.21.15.177:3300/0 conn(0x7ff4c4136a40 0x7ff4c4138e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.678+0000 7ff4cbb9d640 1 --2- 172.21.15.110:0/625487616 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c40a1bf0 0x7ff4c4131810 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.678+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/625487616 >> 172.21.15.110:0/625487616 conn(0x7ff4c4062dd0 msgr2=0x7ff4c40631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.678+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/625487616 shutdown_connections 2024-09-15T04:05:10.693 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.678+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/625487616 wait complete. 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.678+0000 7ff4cbb9d640 10 monclient: build_initial_monmap 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.678+0000 7ff4cbb9d640 10 monclient: monmap: 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 1 Processor -- start 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 1 -- start start 2024-09-15T04:05:10.694 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 10 monclient: init 2024-09-15T04:05:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T04:05:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c409e780 0x7ff4c4141130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4c409e780 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4c40a1bf0 0x7ff4c412e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4c40a1bf0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c4136a40 0x7ff4c4097bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4c4136a40 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c4099dd0 con 0x7ff4c409e780 2024-09-15T04:05:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c409c250 con 0x7ff4c40a1bf0 2024-09-15T04:05:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:10.695 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c412f510 con 0x7ff4c4136a40 2024-09-15T04:05:10.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:10.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4cbb9d640 10 monclient(hunting): authenticate will time out at 1134.323688s 2024-09-15T04:05:10.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9939640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c409e780 0x7ff4c4141130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9939640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c409e780 0x7ff4c4141130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:54712/0 (socket says 172.21.15.110:54712) 2024-09-15T04:05:10.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9939640 1 -- 172.21.15.110:0/82052589 learned_addr learned my addr 172.21.15.110:0/82052589 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:10.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9939640 10 monclient(hunting): get_auth_request con 0x7ff4c409e780 auth_method 0 2024-09-15T04:05:10.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9939640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:10.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9939640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:10.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9939640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:10.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4ca13a640 1 --2- 172.21.15.110:0/82052589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c4136a40 0x7ff4c4097bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4ca13a640 10 monclient(hunting): get_auth_request con 0x7ff4c4136a40 auth_method 0 2024-09-15T04:05:10.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4ca13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:10.696 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4ca13a640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:10.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4ca13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:10.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9939640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:10.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9939640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:10.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9939640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:10.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4ca13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:10.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4ca13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:10.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4ca13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:10.697 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9138640 1 --2- 172.21.15.110:0/82052589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4c40a1bf0 0x7ff4c412e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9138640 10 monclient(hunting): get_auth_request con 0x7ff4c40a1bf0 auth_method 0 2024-09-15T04:05:10.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9138640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:10.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9138640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:10.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.679+0000 7ff4c9138640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:10.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.680+0000 7ff4ca13a640 10 monclient(hunting): handle_auth_done global_id 74219 payload 824 2024-09-15T04:05:10.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.680+0000 7ff4ca13a640 1 -- 172.21.15.110:0/82052589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4c40a1bf0 msgr2=0x7ff4c412e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.680+0000 7ff4ca13a640 1 --2- 172.21.15.110:0/82052589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4c40a1bf0 0x7ff4c412e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.680+0000 7ff4ca13a640 1 -- 172.21.15.110:0/82052589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c409e780 msgr2=0x7ff4c4141130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.680+0000 7ff4ca13a640 1 --2- 172.21.15.110:0/82052589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c409e780 0x7ff4c4141130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.680+0000 7ff4ca13a640 10 monclient: _finish_hunting 0 2024-09-15T04:05:10.698 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.680+0000 7ff4ca13a640 1 monclient: found mon.noname-c 2024-09-15T04:05:10.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.680+0000 7ff4ca13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:10.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.680+0000 7ff4ca13a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:10.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.680+0000 7ff4ca13a640 1 -- 172.21.15.110:0/82052589 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c0009390 con 0x7ff4c4136a40 2024-09-15T04:05:10.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.680+0000 7ff4c9939640 1 --2- 172.21.15.110:0/82052589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c409e780 0x7ff4c4141130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:10.699 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4ca13a640 1 -- 172.21.15.110:0/82052589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c4136a40 msgr2=0x7ff4c4097bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:10.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4ca13a640 1 --2- 172.21.15.110:0/82052589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c4136a40 0x7ff4c4097bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00bad0 tx=0x7ff4bc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:10.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4b9ffb640 1 -- 172.21.15.110:0/82052589 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc00cbc0 con 0x7ff4c4136a40 2024-09-15T04:05:10.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:10.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4b9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:10.700 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4b9ffb640 10 monclient: dump: 2024-09-15T04:05:10.700 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:10.700 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.700 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.700 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.701 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:10.701 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.701 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:10.701 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:10.701 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:10.701 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4b9ffb640 10 monclient: _finish_auth 0 2024-09-15T04:05:10.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:10.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4cbb9d640 5 monclient: authenticate success, global_id 74219 2024-09-15T04:05:10.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4b9ffb640 1 -- 172.21.15.110:0/82052589 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4bc00cd20 con 0x7ff4c4136a40 2024-09-15T04:05:10.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:10.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4cbb9d640 10 monclient: _renew_subs 2024-09-15T04:05:10.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4cbb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:10.701 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4b9ffb640 1 -- 172.21.15.110:0/82052589 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc014660 con 0x7ff4c4136a40 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/82052589 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c409b310 con 0x7ff4c4136a40 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4bbfff640 4 set_mon_vals no callback set 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4b9ffb640 10 monclient: dump: 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:10.702 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:10.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4cbb9d640 10 monclient: _renew_subs 2024-09-15T04:05:10.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4cbb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:10.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.682+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/82052589 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c4099850 con 0x7ff4c4136a40 2024-09-15T04:05:10.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.683+0000 7ff4b9ffb640 1 -- 172.21.15.110:0/82052589 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7ff4bc014a90 con 0x7ff4c4136a40 2024-09-15T04:05:10.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.683+0000 7ff4b9ffb640 1 --2- 172.21.15.110:0/82052589 >> v2:172.21.15.177:6812/28201 conn(0x7ff4a8038700 0x7ff4a803abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:10.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.683+0000 7ff4b9ffb640 1 -- 172.21.15.110:0/82052589 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff4bc04b690 con 0x7ff4c4136a40 2024-09-15T04:05:10.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.683+0000 7ff4cbb9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:10.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.683+0000 7ff4cbb9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:10.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.683+0000 7ff4cbb9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:10.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.683+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/82052589 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c4070950 con 0x7ff4c4136a40 2024-09-15T04:05:10.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.683+0000 7ff4c9939640 1 --2- 172.21.15.110:0/82052589 >> v2:172.21.15.177:6812/28201 conn(0x7ff4a8038700 0x7ff4a803abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:10.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.683+0000 7ff4c9939640 10 monclient: get_auth_request con 0x7ff4a8038700 auth_method 0 2024-09-15T04:05:10.703 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.686+0000 7ff4c9939640 1 --2- 172.21.15.110:0/82052589 >> v2:172.21.15.177:6812/28201 conn(0x7ff4a8038700 0x7ff4a803abb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0009130 tx=0x7ff4c0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:10.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.687+0000 7ff4b9ffb640 1 -- 172.21.15.110:0/82052589 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7ff4bc0112c0 con 0x7ff4c4136a40 2024-09-15T04:05:10.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.687+0000 7ff4b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:10.704 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.687+0000 7ff4b9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:10.839 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.838+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/82052589 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4c409a4e0 con 0x7ff4a8038700 2024-09-15T04:05:10.841 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.840+0000 7ff4b9ffb640 1 -- 172.21.15.110:0/82052589 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+52166 (secure 0 0 0) 0x7ff4c409a4e0 con 0x7ff4a8038700 2024-09-15T04:05:10.841 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T04:05:10.842 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-09-15T04:05:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.842+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/82052589 >> v2:172.21.15.177:6812/28201 conn(0x7ff4a8038700 msgr2=0x7ff4a803abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.842+0000 7ff4cbb9d640 1 --2- 172.21.15.110:0/82052589 >> v2:172.21.15.177:6812/28201 conn(0x7ff4a8038700 0x7ff4a803abb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0009130 tx=0x7ff4c0032000 comp rx=0 tx=0).stop 2024-09-15T04:05:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.842+0000 7ff4cbb9d640 10 monclient: shutdown 2024-09-15T04:05:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.842+0000 7ff4cbb9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.842+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/82052589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c4136a40 msgr2=0x7ff4c4097bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.842+0000 7ff4cbb9d640 1 --2- 172.21.15.110:0/82052589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c4136a40 0x7ff4c4097bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00bad0 tx=0x7ff4bc009780 comp rx=0 tx=0).stop 2024-09-15T04:05:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.842+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/82052589 shutdown_connections 2024-09-15T04:05:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.842+0000 7ff4cbb9d640 1 --2- 172.21.15.110:0/82052589 >> v2:172.21.15.177:6812/28201 conn(0x7ff4a8038700 0x7ff4a803abb0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.843 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.842+0000 7ff4cbb9d640 1 --2- 172.21.15.110:0/82052589 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7ff4c4136a40 0x7ff4c4097bf0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.842+0000 7ff4cbb9d640 1 --2- 172.21.15.110:0/82052589 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff4c40a1bf0 0x7ff4c412e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.842+0000 7ff4cbb9d640 1 --2- 172.21.15.110:0/82052589 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff4c409e780 0x7ff4c4141130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:10.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.842+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/82052589 >> 172.21.15.110:0/82052589 conn(0x7ff4c4062dd0 msgr2=0x7ff4c40a4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:10.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.842+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/82052589 shutdown_connections 2024-09-15T04:05:10.844 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:10.842+0000 7ff4cbb9d640 1 -- 172.21.15.110:0/82052589 wait complete. 2024-09-15T04:05:10.854 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":506,"stamp":"2024-09-15T04:05:10.442960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562077696,"kb_used":4800656,"kb_used_data":557277040,"kb_used_omap":58127,"kb_used_meta":0,"kb_avail":557277040,"statfs":{"total":575567560704,"available":570651688960,"internally_reserved":0,"allocated":570651688960,"data_stored":570651688960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59522189,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6099,"num_objects":-50,"num_object_clones":0,"num_object_copies":-100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-50,"num_write_kb":-50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002350"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":281,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.812288+0000","last_change":"2024-09-15T03:58:44.072431+0000","last_active":"2024-09-15T04:04:47.812288+0000","last_peered":"2024-09-15T04:04:47.812288+0000","last_clean":"2024-09-15T04:04:47.812288+0000","last_became_active":"2024-09-15T03:58:42.157123+0000","last_became_peered":"2024-09-15T03:58:42.157123+0000","last_unstale":"2024-09-15T04:04:47.812288+0000","last_undegraded":"2024-09-15T04:04:47.812288+0000","last_fullsized":"2024-09-15T04:04:47.812288+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:46:37.797943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066518800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":302,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.808620+0000","last_change":"2024-09-15T03:58:44.072012+0000","last_active":"2024-09-15T04:04:47.808620+0000","last_peered":"2024-09-15T04:04:47.808620+0000","last_clean":"2024-09-15T04:04:47.808620+0000","last_became_active":"2024-09-15T03:58:42.069324+0000","last_became_peered":"2024-09-15T03:58:42.069324+0000","last_unstale":"2024-09-15T04:04:47.808620+0000","last_undegraded":"2024-09-15T04:04:47.808620+0000","last_fullsized":"2024-09-15T04:04:47.808620+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:44:04.033148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040535300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":281,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.811753+0000","last_change":"2024-09-15T03:58:44.072166+0000","last_active":"2024-09-15T04:04:47.811753+0000","last_peered":"2024-09-15T04:04:47.811753+0000","last_clean":"2024-09-15T04:04:47.811753+0000","last_became_active":"2024-09-15T03:58:42.155521+0000","last_became_peered":"2024-09-15T03:58:42.155521+0000","last_unstale":"2024-09-15T04:04:47.811753+0000","last_undegraded":"2024-09-15T04:04:47.811753+0000","last_fullsized":"2024-09-15T04:04:47.811753+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:42:01.920640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040527199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":302,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.808414+0000","last_change":"2024-09-15T03:58:44.072137+0000","last_active":"2024-09-15T04:04:47.808414+0000","last_peered":"2024-09-15T04:04:47.808414+0000","last_clean":"2024-09-15T04:04:47.808414+0000","last_became_active":"2024-09-15T03:58:42.155317+0000","last_became_peered":"2024-09-15T03:58:42.155317+0000","last_unstale":"2024-09-15T04:04:47.808414+0000","last_undegraded":"2024-09-15T04:04:47.808414+0000","last_fullsized":"2024-09-15T04:04:47.808414+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:55:28.163584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041321199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":275,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.813708+0000","last_change":"2024-09-15T03:58:44.074749+0000","last_active":"2024-09-15T04:04:47.813708+0000","last_peered":"2024-09-15T04:04:47.813708+0000","last_clean":"2024-09-15T04:04:47.813708+0000","last_became_active":"2024-09-15T03:58:42.071376+0000","last_became_peered":"2024-09-15T03:58:42.071376+0000","last_unstale":"2024-09-15T04:04:47.813708+0000","last_undegraded":"2024-09-15T04:04:47.813708+0000","last_fullsized":"2024-09-15T04:04:47.813708+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:42:52.640034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080084100000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":289,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.809641+0000","last_change":"2024-09-15T03:58:44.071453+0000","last_active":"2024-09-15T04:04:47.809641+0000","last_peered":"2024-09-15T04:04:47.809641+0000","last_clean":"2024-09-15T04:04:47.809641+0000","last_became_active":"2024-09-15T03:58:42.067012+0000","last_became_peered":"2024-09-15T03:58:42.067012+0000","last_unstale":"2024-09-15T04:04:47.809641+0000","last_undegraded":"2024-09-15T04:04:47.809641+0000","last_fullsized":"2024-09-15T04:04:47.809641+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:04:08.843151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040766700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":281,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.811226+0000","last_change":"2024-09-15T03:58:44.072155+0000","last_active":"2024-09-15T04:04:47.811226+0000","last_peered":"2024-09-15T04:04:47.811226+0000","last_clean":"2024-09-15T04:04:47.811226+0000","last_became_active":"2024-09-15T03:58:42.069300+0000","last_became_peered":"2024-09-15T03:58:42.069300+0000","last_unstale":"2024-09-15T04:04:47.811226+0000","last_undegraded":"2024-09-15T04:04:47.811226+0000","last_fullsized":"2024-09-15T04:04:47.811226+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:28:34.591535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036612400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":274,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.814025+0000","last_change":"2024-09-15T03:58:44.074233+0000","last_active":"2024-09-15T04:04:47.814025+0000","last_peered":"2024-09-15T04:04:47.814025+0000","last_clean":"2024-09-15T04:04:47.814025+0000","last_became_active":"2024-09-15T03:58:42.071652+0000","last_became_peered":"2024-09-15T03:58:42.071652+0000","last_unstale":"2024-09-15T04:04:47.814025+0000","last_undegraded":"2024-09-15T04:04:47.814025+0000","last_fullsized":"2024-09-15T04:04:47.814025+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:41.054449+0000","last_clean_scrub_stamp":"2024-09-15T03:58:41.054449+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:31:44.288660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030212199999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":369,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-09-15T04:04:47.811314+0000","last_change":"2024-09-15T03:58:38.057397+0000","last_active":"2024-09-15T04:04:47.811314+0000","last_peered":"2024-09-15T04:04:47.811314+0000","last_clean":"2024-09-15T04:04:47.811314+0000","last_became_active":"2024-09-15T03:58:38.056673+0000","last_became_peered":"2024-09-15T03:58:38.056673+0000","last_unstale":"2024-09-15T04:04:47.811314+0000","last_undegraded":"2024-09-15T04:04:47.811314+0000","last_fullsized":"2024-09-15T04:04:47.811314+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:58:37.036429+0000","last_clean_scrub_stamp":"2024-09-15T03:58:37.036429+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:38:58.792683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799916,"kb_used_data":92879700,"kb_used_omap":9352,"kb_used_meta":0,"kb_avail":92879700,"statfs":{"total":95927926784,"available":95108812800,"internally_reserved":0,"allocated":95108812800,"data_stored":95108812800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9577135,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48399999999999999,"15min":0.64000000000000001},"min":{"1min":0.32800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63800000000000001,"5min":0.70399999999999996,"15min":5.0810000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.50900000000000001,"15min":0.46100000000000002},"min":{"1min":0.33700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":0.51300000000000001}]},{"osd":1,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.54600000000000004,"15min":0.47899999999999998},"min":{"1min":0.26500000000000001,"5min":0.248,"15min":0.248},"max":{"1min":2.6720000000000002,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51700000000000002,"15min":0.46100000000000002},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66200000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.58199999999999996}]},{"osd":2,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.61799999999999999,"5min":0.63900000000000001,"15min":0.71099999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.51500000000000001,"15min":0.73799999999999999},"min":{"1min":0.38600000000000001,"5min":0.314,"15min":0.314},"max":{"1min":2.6320000000000001,"5min":2.6320000000000001,"15min":7.3520000000000003},"last":0.55600000000000005}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.24199999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.497,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.28100000000000003,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.57699999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.26500000000000001}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.38900000000000001,"15min":0.56999999999999995},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.5059999999999998,"5min":2.5059999999999998,"15min":5.1879999999999997},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.377,"5min":0.38200000000000001,"15min":0.57499999999999996},"min":{"1min":0.28399999999999997,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.51800000000000002,"5min":0.60699999999999998,"15min":5.1749999999999998},"last":0.307}]}]},{"osd":4,"up_from":11,"seq":47244640336,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799912,"kb_used_data":92879704,"kb_used_omap":9866,"kb_used_meta":0,"kb_avail":92879704,"statfs":{"total":95927926784,"available":95108816896,"internally_reserved":0,"allocated":95108816896,"data_stored":95108816896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10103386,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.78200000000000003},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.70299999999999996,"5min":1.3959999999999999,"15min":9.9429999999999996},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.51000000000000001,"15min":0.81699999999999995},"min":{"1min":0.26200000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.68500000000000005,"5min":1.272,"15min":11.509},"last":0.54500000000000004}]},{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56299999999999994,"15min":0.53700000000000003},"min":{"1min":0.311,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.76200000000000001,"5min":2.867,"15min":2.867},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56999999999999995,"15min":0.57699999999999996},"min":{"1min":0.33400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":1.7689999999999999,"5min":3.923,"15min":3.923},"last":0.47499999999999998}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51800000000000002,"15min":0.496},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69999999999999996,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.71799999999999997},"min":{"1min":0.35899999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.73999999999999999,"5min":1.3540000000000001,"15min":7.5140000000000002},"last":0.56299999999999994}]},{"osd":3,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.40000000000000002,"15min":0.39200000000000002},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.53300000000000003,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41999999999999998,"15min":0.42699999999999999},"min":{"1min":0.26600000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.55200000000000005,"5min":2.339,"15min":2.339},"last":0.48799999999999999}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44700000000000001,"15min":0.73599999999999999},"min":{"1min":0.33500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.5,"5min":3.7080000000000002,"15min":11.814},"last":0.505},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.438,"15min":0.42599999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.53300000000000003,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.42999999999999999}]}]},{"osd":3,"up_from":11,"seq":47244640336,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":800500,"kb_used_data":92879116,"kb_used_omap":9496,"kb_used_meta":0,"kb_avail":92879116,"statfs":{"total":95927926784,"available":95108214784,"internally_reserved":0,"allocated":95108214784,"data_stored":95108214784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9724376,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.51600000000000001,"15min":0.48699999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.19900000000000001},"max":{"1min":0.63600000000000001,"5min":3.1429999999999998,"15min":3.1429999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48699999999999999,"15min":7.2050000000000001},"min":{"1min":0.33700000000000002,"5min":0.27900000000000003,"15min":0.23499999999999999},"max":{"1min":0.73699999999999999,"5min":0.76200000000000001,"15min":192.55600000000001},"last":0.35299999999999998}]},{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52100000000000002,"15min":7.2519999999999998},"min":{"1min":0.32300000000000001,"5min":0.30399999999999999,"15min":0.22600000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":192.50899999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55200000000000005,"15min":0.53200000000000003},"min":{"1min":0.32500000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.76700000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.44}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.501,"15min":0.64500000000000002},"min":{"1min":0.35399999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.624,"5min":0.79500000000000004,"15min":5.0419999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.73199999999999998},"min":{"1min":0.40100000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":0.77700000000000002,"15min":7.2930000000000001},"last":0.372}]},{"osd":4,"last update":"Sun Sep 15 04:04:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.36299999999999999,"15min":0.35499999999999998},"min":{"1min":0.223,"5min":0.222,"15min":0.222},"max":{"1min":0.46000000000000002,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.41299999999999998,"15min":0.38700000000000001},"min":{"1min":0.34499999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67300000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.45300000000000001}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39200000000000002},"min":{"1min":0.27100000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.625,"5min":0.67800000000000005,"15min":0.69999999999999996},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.39400000000000002,"15min":0.40400000000000003},"min":{"1min":0.30399999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.5,"5min":0.625,"15min":0.625},"last":0.42199999999999999}]}]},{"osd":2,"up_from":11,"seq":47244640336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799916,"kb_used_data":92879700,"kb_used_omap":9704,"kb_used_meta":0,"kb_avail":92879700,"statfs":{"total":95927926784,"available":95108812800,"internally_reserved":0,"allocated":95108812800,"data_stored":95108812800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9937685,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.316,"15min":0.34000000000000002},"min":{"1min":0.20699999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.47699999999999998,"5min":0.502,"15min":0.57699999999999996},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.33000000000000002,"15min":0.36399999999999999},"min":{"1min":0.223,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.44500000000000001,"5min":0.56999999999999995,"15min":0.65100000000000002},"last":0.21199999999999999}]},{"osd":1,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.34599999999999997},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48999999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.38,"5min":0.35899999999999999,"15min":0.372},"min":{"1min":0.28499999999999998,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54400000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.25900000000000001}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.67100000000000004,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72299999999999998,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.52200000000000002}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.66200000000000003,"5min":0.745,"15min":0.745},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54100000000000004,"15min":0.63500000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.754,"5min":1.208,"15min":3.1339999999999999},"last":0.502}]},{"osd":5,"last update":"Sun Sep 15 04:04:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.35999999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.65400000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73199999999999998,"5min":0.77800000000000002,"15min":2.9500000000000002},"last":0.54900000000000004}]}]},{"osd":1,"up_from":11,"seq":47244640336,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799920,"kb_used_data":92879696,"kb_used_omap":9674,"kb_used_meta":0,"kb_avail":92879696,"statfs":{"total":95927926784,"available":95108808704,"internally_reserved":0,"allocated":95108808704,"data_stored":95108808704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9906960,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34499999999999997,"15min":0.34300000000000003},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.52700000000000002,"5min":1.101,"15min":1.101},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.35299999999999998,"15min":0.35099999999999998},"min":{"1min":0.26400000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.497,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.376,"15min":0.36699999999999999},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":0.55600000000000005,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.22},{"interface":"front","average":{"1min":0.374,"5min":0.36199999999999999,"15min":0.374},"min":{"1min":0.26300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.5,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.248}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52200000000000002,"15min":0.54800000000000004},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.247},"max":{"1min":0.65400000000000003,"5min":1.087,"15min":1.087},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.54900000000000004},"min":{"1min":0.35999999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68500000000000005,"5min":1.032,"15min":1.032},"last":0.47499999999999998}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.57699999999999996,"15min":0.56100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66400000000000003,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":0.495},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.623,"15min":0.59399999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68300000000000005,"5min":4.2930000000000001,"15min":4.2930000000000001},"last":0.51400000000000001}]},{"osd":5,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.33000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.72899999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.36899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70899999999999996,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.58299999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640336,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":800492,"kb_used_data":92879124,"kb_used_omap":10031,"kb_used_meta":0,"kb_avail":92879124,"statfs":{"total":95927926784,"available":95108222976,"internally_reserved":0,"allocated":95108222976,"data_stored":95108222976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10272647,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"min":{"1min":0.246,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.49099999999999999,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.35999999999999999},"min":{"1min":0.309,"5min":0.193,"15min":0.193},"max":{"1min":0.496,"5min":0.504,"15min":0.71199999999999997},"last":0.29699999999999999}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.34799999999999998,"15min":0.376},"min":{"1min":0.32400000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52000000000000002,"5min":0.67500000000000004,"15min":0.78300000000000003},"last":0.313},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.38800000000000001},"min":{"1min":0.29199999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.54100000000000004,"5min":0.65200000000000002,"15min":0.76000000000000001},"last":0.33700000000000002}]},{"osd":3,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.43099999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.80100000000000005,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72299999999999998,"5min":0.85599999999999998,"15min":0.88200000000000001},"last":0.57399999999999995}]},{"osd":4,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.46300000000000002,"5min":0.30099999999999999,"15min":0.29599999999999999},"max":{"1min":0.84599999999999997,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.57199999999999995,"15min":0.59599999999999997},"min":{"1min":0.53100000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.59599999999999997}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52800000000000002,"15min":0.54800000000000004},"min":{"1min":0.44500000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.56200000000000006,"15min":0.54800000000000004},"min":{"1min":0.47399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76000000000000001,"5min":1.044,"15min":1.044},"last":0.51800000000000002}]}]}],"pool_statfs":[]}} 2024-09-15T04:05:10.858 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-15T03:58:41.054449+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=58, tm_sec=41, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=4, tm_min=5, tm_sec=7, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T04:05:10.859 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-15T03:58:41.054449+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=58, tm_sec=41, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=4, tm_min=5, tm_sec=7, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T04:05:10.859 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-15T03:58:41.054449+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=58, tm_sec=41, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=4, tm_min=5, tm_sec=7, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T04:05:10.859 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-15T03:58:41.054449+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=58, tm_sec=41, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=4, tm_min=5, tm_sec=7, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T04:05:10.859 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-15T03:58:41.054449+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=58, tm_sec=41, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=4, tm_min=5, tm_sec=7, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T04:05:10.859 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-15T03:58:41.054449+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=58, tm_sec=41, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=4, tm_min=5, tm_sec=7, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T04:05:10.859 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-15T03:58:41.054449+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=58, tm_sec=41, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=4, tm_min=5, tm_sec=7, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T04:05:10.859 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-15T03:58:41.054449+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=58, tm_sec=41, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=4, tm_min=5, tm_sec=7, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T04:05:10.859 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-15T03:58:37.036429+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=58, tm_sec=37, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=4, tm_min=5, tm_sec=7, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T04:05:10.860 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T04:05:30.861 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T04:05:30.965 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.965+0000 7efda6a20640 10 monclient: get_monmap_and_config 2024-09-15T04:05:30.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.965+0000 7efda6a20640 10 monclient: build_initial_monmap 2024-09-15T04:05:30.966 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.965+0000 7efda6a20640 10 monclient: monmap: 2024-09-15T04:05:30.966 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:30.966 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:30.966 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:30.966247+0000 2024-09-15T04:05:30.966 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:30.966247+0000 2024-09-15T04:05:30.966 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:30.966 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:30.966 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:30.966 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:30.966 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:30.966 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:30.966 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:30.967 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:30.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 1 Processor -- start 2024-09-15T04:05:30.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 1 -- start start 2024-09-15T04:05:30.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 10 monclient: init 2024-09-15T04:05:30.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:30.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-15T04:05:30.969 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7efda00a1bf0 0x7efda0131810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 10 monclient(hunting): picked mon.noname-c con 0x7efda00a1bf0 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009e780 0x7efda009eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 10 monclient(hunting): picked mon.noname-b con 0x7efda009e780 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efda009f760 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda009c9b0 con 0x7efda009f760 2024-09-15T04:05:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda00a0c20 con 0x7efda00a1bf0 2024-09-15T04:05:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda009cc90 con 0x7efda009e780 2024-09-15T04:05:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda6a20640 10 monclient(hunting): authenticate will time out at 1154.610652s 2024-09-15T04:05:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009e780 0x7efda009eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009e780 0x7efda009eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50180/0 (socket says 172.21.15.110:50180) 2024-09-15T04:05:30.970 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efda4fbd640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7efda009f760 0x7efda009fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36812/0 (socket says 172.21.15.110:36812) 2024-09-15T04:05:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9f7fe640 1 -- 172.21.15.110:0/1981509117 learned_addr learned my addr 172.21.15.110:0/1981509117 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9f7fe640 10 monclient(hunting): get_auth_request con 0x7efda009e780 auth_method 0 2024-09-15T04:05:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9ffff640 1 --2- 172.21.15.110:0/1981509117 >> v2:172.21.15.177:3300/0 conn(0x7efda00a1bf0 0x7efda0131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9ffff640 10 monclient(hunting): get_auth_request con 0x7efda00a1bf0 auth_method 0 2024-09-15T04:05:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:30.971 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.966+0000 7efd9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.967+0000 7efd9f7fe640 10 monclient(hunting): handle_auth_done global_id 74222 payload 274 2024-09-15T04:05:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.967+0000 7efd9f7fe640 1 -- 172.21.15.110:0/1981509117 >> v2:172.21.15.177:3300/0 conn(0x7efda00a1bf0 msgr2=0x7efda0131810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.967+0000 7efd9f7fe640 1 --2- 172.21.15.110:0/1981509117 >> v2:172.21.15.177:3300/0 conn(0x7efda00a1bf0 0x7efda0131810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.967+0000 7efd9f7fe640 1 -- 172.21.15.110:0/1981509117 >> v1:172.21.15.110:6789/0 conn(0x7efda009f760 legacy=0x7efda009fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.967+0000 7efd9f7fe640 10 monclient: _finish_hunting 0 2024-09-15T04:05:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.967+0000 7efd9f7fe640 1 monclient: found mon.noname-b 2024-09-15T04:05:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.967+0000 7efd9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.967+0000 7efd9f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3301/0 2024-09-15T04:05:30.972 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.967+0000 7efd9f7fe640 1 -- 172.21.15.110:0/1981509117 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda00a1fc0 con 0x7efda009e780 2024-09-15T04:05:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.969+0000 7efd9f7fe640 1 -- 172.21.15.110:0/1981509117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009e780 msgr2=0x7efda009eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.969+0000 7efd9f7fe640 1 --2- 172.21.15.110:0/1981509117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009e780 0x7efda009eb50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efd94009910 tx=0x7efd940091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54369afc49195f78 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efd9effd640 1 -- 172.21.15.110:0/1981509117 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd94032440 con 0x7efda009e780 2024-09-15T04:05:30.981 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efd9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efd9effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T04:05:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efd9effd640 10 monclient: dump: 2024-09-15T04:05:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:30.982 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:30.983 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efd9effd640 10 monclient: _finish_auth 0 2024-09-15T04:05:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efd9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efd9effd640 1 -- 172.21.15.110:0/1981509117 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd940325a0 con 0x7efda009e780 2024-09-15T04:05:30.983 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efda6a20640 5 monclient: authenticate success, global_id 74222 2024-09-15T04:05:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efda6a20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efd9effd640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efda6a20640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efd9effd640 1 -- 172.21.15.110:0/1981509117 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd94032940 con 0x7efda009e780 2024-09-15T04:05:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efda6a20640 4 set_mon_vals no callback set 2024-09-15T04:05:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efd9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efd9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efd9effd640 10 monclient: dump: 2024-09-15T04:05:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:30.984 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:30.985 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efda6a20640 10 monclient: shutdown 2024-09-15T04:05:30.985 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efda6a20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:30.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efda6a20640 1 -- 172.21.15.110:0/1981509117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009e780 msgr2=0x7efda009eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:30.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efda6a20640 1 --2- 172.21.15.110:0/1981509117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009e780 0x7efda009eb50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efd94009910 tx=0x7efd940091d0 comp rx=0 tx=0).stop 2024-09-15T04:05:30.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efda6a20640 1 -- 172.21.15.110:0/1981509117 shutdown_connections 2024-09-15T04:05:30.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efda6a20640 1 --2- 172.21.15.110:0/1981509117 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009e780 0x7efda009eb50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:30.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efda6a20640 1 --2- 172.21.15.110:0/1981509117 >> v2:172.21.15.177:3300/0 conn(0x7efda00a1bf0 0x7efda0131810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:30.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efda6a20640 1 -- 172.21.15.110:0/1981509117 >> 172.21.15.110:0/1981509117 conn(0x7efda0062dd0 msgr2=0x7efda00631d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:30.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efda6a20640 1 -- 172.21.15.110:0/1981509117 shutdown_connections 2024-09-15T04:05:30.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.970+0000 7efda6a20640 1 -- 172.21.15.110:0/1981509117 wait complete. 2024-09-15T04:05:30.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.971+0000 7efda6a20640 10 monclient: build_initial_monmap 2024-09-15T04:05:30.986 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.971+0000 7efda6a20640 10 monclient: monmap: 2024-09-15T04:05:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:30.987 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.971+0000 7efda6a20640 1 Processor -- start 2024-09-15T04:05:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.971+0000 7efda6a20640 1 -- start start 2024-09-15T04:05:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.971+0000 7efda6a20640 10 monclient: init 2024-09-15T04:05:30.987 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T04:05:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efda009e780 0x7efda0099c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 10 monclient(hunting): picked mon.noname-b con 0x7efda009e780 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009f760 0x7efda0130a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 10 monclient(hunting): picked mon.noname-c con 0x7efda009f760 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efda00a1bf0 0x7efda0130f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 10 monclient(hunting): picked mon.noname-a con 0x7efda00a1bf0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:30.988 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda009a840 con 0x7efda00a1bf0 2024-09-15T04:05:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda009c250 con 0x7efda009e780 2024-09-15T04:05:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda009ae00 con 0x7efda009f760 2024-09-15T04:05:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda6a20640 10 monclient(hunting): authenticate will time out at 1154.616145s 2024-09-15T04:05:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009f760 0x7efda0130a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009f760 0x7efda0130a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3301/0 says I am v2:172.21.15.110:50190/0 (socket says 172.21.15.110:50190) 2024-09-15T04:05:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 1 -- 172.21.15.110:0/2814657830 learned_addr learned my addr 172.21.15.110:0/2814657830 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:30.989 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 10 monclient(hunting): get_auth_request con 0x7efda009f760 auth_method 0 2024-09-15T04:05:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda4fbd640 1 --2- 172.21.15.110:0/2814657830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efda00a1bf0 0x7efda0130f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda4fbd640 10 monclient(hunting): get_auth_request con 0x7efda00a1bf0 auth_method 0 2024-09-15T04:05:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda4fbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda4fbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda4fbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:30.990 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9ffff640 1 --2- 172.21.15.110:0/2814657830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efda009e780 0x7efda0099c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda4fbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda4fbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda4fbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9ffff640 10 monclient(hunting): get_auth_request con 0x7efda009e780 auth_method 0 2024-09-15T04:05:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:30.991 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 10 monclient(hunting): handle_auth_done global_id 74225 payload 824 2024-09-15T04:05:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 1 -- 172.21.15.110:0/2814657830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efda009e780 msgr2=0x7efda0099c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 1 --2- 172.21.15.110:0/2814657830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efda009e780 0x7efda0099c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 1 -- 172.21.15.110:0/2814657830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efda00a1bf0 msgr2=0x7efda0130f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 1 --2- 172.21.15.110:0/2814657830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efda00a1bf0 0x7efda0130f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 10 monclient: _finish_hunting 0 2024-09-15T04:05:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 1 monclient: found mon.noname-c 2024-09-15T04:05:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:30.992 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efd9f7fe640 1 -- 172.21.15.110:0/2814657830 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd94009450 con 0x7efda009f760 2024-09-15T04:05:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.972+0000 7efda4fbd640 1 --2- 172.21.15.110:0/2814657830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efda00a1bf0 0x7efda0130f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.974+0000 7efd9f7fe640 1 -- 172.21.15.110:0/2814657830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009f760 msgr2=0x7efda0130a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.974+0000 7efd9f7fe640 1 --2- 172.21.15.110:0/2814657830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009f760 0x7efda0130a00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efd94009970 tx=0x7efd94032000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efd87fff640 1 -- 172.21.15.110:0/2814657830 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd94031a40 con 0x7efda009f760 2024-09-15T04:05:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efd87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efd87fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efd87fff640 10 monclient: dump: 2024-09-15T04:05:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:30.993 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:30.994 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efd87fff640 10 monclient: _finish_auth 0 2024-09-15T04:05:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efd87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efda6a20640 5 monclient: authenticate success, global_id 74225 2024-09-15T04:05:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efd87fff640 1 -- 172.21.15.110:0/2814657830 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd94031bc0 con 0x7efda009f760 2024-09-15T04:05:30.994 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efda6a20640 10 monclient: _renew_subs 2024-09-15T04:05:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efda6a20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efda6a20640 1 -- 172.21.15.110:0/2814657830 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda011db20 con 0x7efda009f760 2024-09-15T04:05:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efd87fff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efd87fff640 1 -- 172.21.15.110:0/2814657830 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9403d440 con 0x7efda009f760 2024-09-15T04:05:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efd87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efd87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efd87fff640 10 monclient: dump: 2024-09-15T04:05:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:30.995 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:30.996 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efda6a20640 10 monclient: _renew_subs 2024-09-15T04:05:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efd9e7fc640 4 set_mon_vals no callback set 2024-09-15T04:05:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efda6a20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.975+0000 7efda6a20640 1 -- 172.21.15.110:0/2814657830 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda011cc30 con 0x7efda009f760 2024-09-15T04:05:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.976+0000 7efd87fff640 1 -- 172.21.15.110:0/2814657830 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7efd94039040 con 0x7efda009f760 2024-09-15T04:05:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.976+0000 7efda6a20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.976+0000 7efda6a20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.976+0000 7efda6a20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.976+0000 7efda6a20640 1 -- 172.21.15.110:0/2814657830 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda00a3f00 con 0x7efda009f760 2024-09-15T04:05:30.996 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.976+0000 7efd87fff640 1 --2- 172.21.15.110:0/2814657830 >> v2:172.21.15.177:6812/28201 conn(0x7efd8c038700 0x7efd8c03abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.976+0000 7efd87fff640 1 -- 172.21.15.110:0/2814657830 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7efd9406b2a0 con 0x7efda009f760 2024-09-15T04:05:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.976+0000 7efd9ffff640 1 --2- 172.21.15.110:0/2814657830 >> v2:172.21.15.177:6812/28201 conn(0x7efd8c038700 0x7efd8c03abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.976+0000 7efd9ffff640 10 monclient: get_auth_request con 0x7efd8c038700 auth_method 0 2024-09-15T04:05:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.979+0000 7efd9ffff640 1 --2- 172.21.15.110:0/2814657830 >> v2:172.21.15.177:6812/28201 conn(0x7efd8c038700 0x7efd8c03abb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efd90007ba0 tx=0x7efd90008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.980+0000 7efd87fff640 1 -- 172.21.15.110:0/2814657830 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7efda00a3f00 con 0x7efda009f760 2024-09-15T04:05:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.980+0000 7efd87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:30.997 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:30.980+0000 7efd87fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:31.134 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.134+0000 7efda6a20640 1 -- 172.21.15.110:0/2814657830 --> v2:172.21.15.177:6812/28201 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efda00a3f00 con 0x7efd8c038700 2024-09-15T04:05:31.137 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.137+0000 7efd87fff640 1 -- 172.21.15.110:0/2814657830 <== mgr.4105 v2:172.21.15.177:6812/28201 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+52347 (secure 0 0 0) 0x7efda00a3f00 con 0x7efd8c038700 2024-09-15T04:05:31.137 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-09-15T04:05:31.138 INFO:teuthology.orchestra.run.smithi110.stderr:dumped all 2024-09-15T04:05:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.138+0000 7efda6a20640 1 -- 172.21.15.110:0/2814657830 >> v2:172.21.15.177:6812/28201 conn(0x7efd8c038700 msgr2=0x7efd8c03abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.138+0000 7efda6a20640 1 --2- 172.21.15.110:0/2814657830 >> v2:172.21.15.177:6812/28201 conn(0x7efd8c038700 0x7efd8c03abb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efd90007ba0 tx=0x7efd90008040 comp rx=0 tx=0).stop 2024-09-15T04:05:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.138+0000 7efda6a20640 10 monclient: shutdown 2024-09-15T04:05:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.138+0000 7efda6a20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:31.139 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.138+0000 7efda6a20640 1 -- 172.21.15.110:0/2814657830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009f760 msgr2=0x7efda0130a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.138+0000 7efda6a20640 1 --2- 172.21.15.110:0/2814657830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009f760 0x7efda0130a00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efd94009970 tx=0x7efd94032000 comp rx=0 tx=0).stop 2024-09-15T04:05:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.138+0000 7efda6a20640 1 -- 172.21.15.110:0/2814657830 shutdown_connections 2024-09-15T04:05:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.138+0000 7efda6a20640 1 --2- 172.21.15.110:0/2814657830 >> v2:172.21.15.177:6812/28201 conn(0x7efd8c038700 0x7efd8c03abb0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.138+0000 7efda6a20640 1 --2- 172.21.15.110:0/2814657830 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7efda00a1bf0 0x7efda0130f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.138+0000 7efda6a20640 1 --2- 172.21.15.110:0/2814657830 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7efda009f760 0x7efda0130a00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.138+0000 7efda6a20640 1 --2- 172.21.15.110:0/2814657830 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7efda009e780 0x7efda0099c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.138+0000 7efda6a20640 1 -- 172.21.15.110:0/2814657830 >> 172.21.15.110:0/2814657830 conn(0x7efda0062dd0 msgr2=0x7efda00a37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.138+0000 7efda6a20640 1 -- 172.21.15.110:0/2814657830 shutdown_connections 2024-09-15T04:05:31.140 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.138+0000 7efda6a20640 1 -- 172.21.15.110:0/2814657830 wait complete. 2024-09-15T04:05:31.150 INFO:teuthology.orchestra.run.smithi110.stdout:{"pg_ready":true,"pg_map":{"version":516,"stamp":"2024-09-15T04:05:30.447370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562077696,"kb_used":4800656,"kb_used_data":557277040,"kb_used_omap":58157,"kb_used_meta":0,"kb_avail":557277040,"statfs":{"total":575567560704,"available":570651688960,"internally_reserved":0,"allocated":570651688960,"data_stored":570651688960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59553438,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002640"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":292,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-15T04:05:16.648416+0000","last_change":"2024-09-15T04:05:16.648393+0000","last_active":"2024-09-15T04:05:16.648416+0000","last_peered":"2024-09-15T04:05:16.648416+0000","last_clean":"2024-09-15T04:05:16.648416+0000","last_became_active":"2024-09-15T03:58:42.157123+0000","last_became_peered":"2024-09-15T03:58:42.157123+0000","last_unstale":"2024-09-15T04:05:16.648416+0000","last_undegraded":"2024-09-15T04:05:16.648416+0000","last_fullsized":"2024-09-15T04:05:16.648416+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T04:05:16.648336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T04:05:16.648336+0000","last_clean_scrub_stamp":"2024-09-15T04:05:16.648336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:19:23.517747+0000","scrub_duration":0.001392903,"objects_trimmed":0,"snaptrim_duration":0.00066518800000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":313,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-15T04:05:12.112261+0000","last_change":"2024-09-15T04:05:12.112235+0000","last_active":"2024-09-15T04:05:12.112261+0000","last_peered":"2024-09-15T04:05:12.112261+0000","last_clean":"2024-09-15T04:05:12.112261+0000","last_became_active":"2024-09-15T03:58:42.069324+0000","last_became_peered":"2024-09-15T03:58:42.069324+0000","last_unstale":"2024-09-15T04:05:12.112261+0000","last_undegraded":"2024-09-15T04:05:12.112261+0000","last_fullsized":"2024-09-15T04:05:12.112261+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T04:05:12.112174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T04:05:12.112174+0000","last_clean_scrub_stamp":"2024-09-15T04:05:12.112174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:40:39.320603+0000","scrub_duration":0.0018218349999999999,"objects_trimmed":0,"snaptrim_duration":0.00040535300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":292,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-15T04:05:19.672354+0000","last_change":"2024-09-15T04:05:19.672331+0000","last_active":"2024-09-15T04:05:19.672354+0000","last_peered":"2024-09-15T04:05:19.672354+0000","last_clean":"2024-09-15T04:05:19.672354+0000","last_became_active":"2024-09-15T03:58:42.155521+0000","last_became_peered":"2024-09-15T03:58:42.155521+0000","last_unstale":"2024-09-15T04:05:19.672354+0000","last_undegraded":"2024-09-15T04:05:19.672354+0000","last_fullsized":"2024-09-15T04:05:19.672354+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T04:05:19.672239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T04:05:19.672239+0000","last_clean_scrub_stamp":"2024-09-15T04:05:19.672239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:22:26.625160+0000","scrub_duration":0.0018457289999999999,"objects_trimmed":0,"snaptrim_duration":0.00040527199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":313,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-15T04:05:13.127386+0000","last_change":"2024-09-15T04:05:13.127342+0000","last_active":"2024-09-15T04:05:13.127386+0000","last_peered":"2024-09-15T04:05:13.127386+0000","last_clean":"2024-09-15T04:05:13.127386+0000","last_became_active":"2024-09-15T03:58:42.155317+0000","last_became_peered":"2024-09-15T03:58:42.155317+0000","last_unstale":"2024-09-15T04:05:13.127386+0000","last_undegraded":"2024-09-15T04:05:13.127386+0000","last_fullsized":"2024-09-15T04:05:13.127386+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T04:05:13.127277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T04:05:13.127277+0000","last_clean_scrub_stamp":"2024-09-15T04:05:13.127277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:59:02.488239+0000","scrub_duration":0.001424395,"objects_trimmed":0,"snaptrim_duration":0.00041321199999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"315'3","reported_seq":287,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-15T04:05:12.398883+0000","last_change":"2024-09-15T04:05:12.398847+0000","last_active":"2024-09-15T04:05:12.398883+0000","last_peered":"2024-09-15T04:05:12.398883+0000","last_clean":"2024-09-15T04:05:12.398883+0000","last_became_active":"2024-09-15T03:58:42.071376+0000","last_became_peered":"2024-09-15T03:58:42.071376+0000","last_unstale":"2024-09-15T04:05:12.398883+0000","last_undegraded":"2024-09-15T04:05:12.398883+0000","last_fullsized":"2024-09-15T04:05:12.398883+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'3","last_scrub_stamp":"2024-09-15T04:05:12.398794+0000","last_deep_scrub":"315'3","last_deep_scrub_stamp":"2024-09-15T04:05:12.398794+0000","last_clean_scrub_stamp":"2024-09-15T04:05:12.398794+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:01:29.725654+0000","scrub_duration":0.0028457000000000001,"objects_trimmed":0,"snaptrim_duration":0.00080084100000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":300,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-15T04:05:09.773725+0000","last_change":"2024-09-15T04:05:09.773705+0000","last_active":"2024-09-15T04:05:09.773725+0000","last_peered":"2024-09-15T04:05:09.773725+0000","last_clean":"2024-09-15T04:05:09.773725+0000","last_became_active":"2024-09-15T03:58:42.067012+0000","last_became_peered":"2024-09-15T03:58:42.067012+0000","last_unstale":"2024-09-15T04:05:09.773725+0000","last_undegraded":"2024-09-15T04:05:09.773725+0000","last_fullsized":"2024-09-15T04:05:09.773725+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T04:05:09.773656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T04:05:09.773656+0000","last_clean_scrub_stamp":"2024-09-15T04:05:09.773656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:40:15.493741+0000","scrub_duration":0.001310762,"objects_trimmed":0,"snaptrim_duration":0.00040766700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":292,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-15T04:05:15.644511+0000","last_change":"2024-09-15T04:05:15.644456+0000","last_active":"2024-09-15T04:05:15.644511+0000","last_peered":"2024-09-15T04:05:15.644511+0000","last_clean":"2024-09-15T04:05:15.644511+0000","last_became_active":"2024-09-15T03:58:42.069300+0000","last_became_peered":"2024-09-15T03:58:42.069300+0000","last_unstale":"2024-09-15T04:05:15.644511+0000","last_undegraded":"2024-09-15T04:05:15.644511+0000","last_fullsized":"2024-09-15T04:05:15.644511+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T04:05:15.644383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T04:05:15.644383+0000","last_clean_scrub_stamp":"2024-09-15T04:05:15.644383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:26:25.190156+0000","scrub_duration":0.0018971560000000001,"objects_trimmed":0,"snaptrim_duration":0.00036612400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"315'2","reported_seq":286,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-15T04:05:11.413513+0000","last_change":"2024-09-15T04:05:11.413487+0000","last_active":"2024-09-15T04:05:11.413513+0000","last_peered":"2024-09-15T04:05:11.413513+0000","last_clean":"2024-09-15T04:05:11.413513+0000","last_became_active":"2024-09-15T03:58:42.071652+0000","last_became_peered":"2024-09-15T03:58:42.071652+0000","last_unstale":"2024-09-15T04:05:11.413513+0000","last_undegraded":"2024-09-15T04:05:11.413513+0000","last_fullsized":"2024-09-15T04:05:11.413513+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'2","last_scrub_stamp":"2024-09-15T04:05:11.413426+0000","last_deep_scrub":"315'2","last_deep_scrub_stamp":"2024-09-15T04:05:11.413426+0000","last_clean_scrub_stamp":"2024-09-15T04:05:11.413426+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:08:34.037477+0000","scrub_duration":0.0043796299999999998,"objects_trimmed":0,"snaptrim_duration":0.00030212199999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"315'74","reported_seq":382,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-09-15T04:05:14.668578+0000","last_change":"2024-09-15T04:05:14.668539+0000","last_active":"2024-09-15T04:05:14.668578+0000","last_peered":"2024-09-15T04:05:14.668578+0000","last_clean":"2024-09-15T04:05:14.668578+0000","last_became_active":"2024-09-15T03:58:38.056673+0000","last_became_peered":"2024-09-15T03:58:38.056673+0000","last_unstale":"2024-09-15T04:05:14.668578+0000","last_undegraded":"2024-09-15T04:05:14.668578+0000","last_fullsized":"2024-09-15T04:05:14.668578+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'74","last_scrub_stamp":"2024-09-15T04:05:14.668482+0000","last_deep_scrub":"315'74","last_deep_scrub_stamp":"2024-09-15T04:05:14.668482+0000","last_clean_scrub_stamp":"2024-09-15T04:05:14.668482+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:38:50.316098+0000","scrub_duration":0.0046161020000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640341,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799916,"kb_used_data":92879700,"kb_used_omap":9357,"kb_used_meta":0,"kb_avail":92879700,"statfs":{"total":95927926784,"available":95108812800,"internally_reserved":0,"allocated":95108812800,"data_stored":95108812800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9582206,"internal_metadata":0},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48399999999999999,"15min":0.64000000000000001},"min":{"1min":0.32800000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.63800000000000001,"5min":0.70399999999999996,"15min":5.0810000000000004},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.50900000000000001,"15min":0.46100000000000002},"min":{"1min":0.33700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":0.41799999999999998}]},{"osd":1,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.54600000000000004,"15min":0.47899999999999998},"min":{"1min":0.26500000000000001,"5min":0.248,"15min":0.248},"max":{"1min":2.6720000000000002,"5min":2.6720000000000002,"15min":2.6720000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51700000000000002,"15min":0.46100000000000002},"min":{"1min":0.315,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66200000000000003,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.39300000000000002}]},{"osd":2,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.61799999999999999,"5min":0.63900000000000001,"15min":0.71099999999999997},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.51500000000000001,"15min":0.73799999999999999},"min":{"1min":0.38600000000000001,"5min":0.314,"15min":0.314},"max":{"1min":2.6320000000000001,"5min":2.6320000000000001,"15min":7.3520000000000003},"last":0.377}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.24199999999999999,"5min":0.219,"15min":0.219},"max":{"1min":0.497,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.28100000000000003,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.57699999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.28699999999999998}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.38900000000000001,"15min":0.56999999999999995},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.5059999999999998,"5min":2.5059999999999998,"15min":5.1879999999999997},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.38200000000000001,"15min":0.57499999999999996},"min":{"1min":0.28399999999999997,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.51800000000000002,"5min":0.60699999999999998,"15min":5.1749999999999998},"last":0.32200000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640340,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799912,"kb_used_data":92879704,"kb_used_omap":9867,"kb_used_meta":0,"kb_avail":92879704,"statfs":{"total":95927926784,"available":95108816896,"internally_reserved":0,"allocated":95108816896,"data_stored":95108816896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10104676,"internal_metadata":0},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51400000000000001,"15min":0.78200000000000003},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.70299999999999996,"5min":1.3959999999999999,"15min":9.9429999999999996},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.51000000000000001,"15min":0.81699999999999995},"min":{"1min":0.26200000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.68500000000000005,"5min":1.272,"15min":11.509},"last":0.67100000000000004}]},{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56299999999999994,"15min":0.53700000000000003},"min":{"1min":0.311,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.76200000000000001,"5min":2.867,"15min":2.867},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56999999999999995,"15min":0.57699999999999996},"min":{"1min":0.33400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":1.7689999999999999,"5min":3.923,"15min":3.923},"last":0.58799999999999997}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51800000000000002,"15min":0.496},"min":{"1min":0.35799999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.69999999999999996,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.71799999999999997},"min":{"1min":0.35899999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.73999999999999999,"5min":1.3540000000000001,"15min":7.5140000000000002},"last":0.64900000000000002}]},{"osd":3,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.40000000000000002,"15min":0.39200000000000002},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.53300000000000003,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.41999999999999998,"15min":0.42699999999999999},"min":{"1min":0.26600000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.55200000000000005,"5min":2.339,"15min":2.339},"last":0.51900000000000002}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44700000000000001,"15min":0.73599999999999999},"min":{"1min":0.33500000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.5,"5min":3.7080000000000002,"15min":11.814},"last":0.496},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.438,"15min":0.42599999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.53300000000000003,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.54400000000000004}]}]},{"osd":3,"up_from":11,"seq":47244640340,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":800500,"kb_used_data":92879116,"kb_used_omap":9504,"kb_used_meta":0,"kb_avail":92879116,"statfs":{"total":95927926784,"available":95108214784,"internally_reserved":0,"allocated":95108214784,"data_stored":95108214784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9732315,"internal_metadata":0},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.51600000000000001,"15min":0.48699999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.19900000000000001},"max":{"1min":0.63600000000000001,"5min":3.1429999999999998,"15min":3.1429999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48699999999999999,"15min":7.2050000000000001},"min":{"1min":0.33700000000000002,"5min":0.27900000000000003,"15min":0.23499999999999999},"max":{"1min":0.73699999999999999,"5min":0.76200000000000001,"15min":192.55600000000001},"last":0.38200000000000001}]},{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52100000000000002,"15min":7.2519999999999998},"min":{"1min":0.32300000000000001,"5min":0.30399999999999999,"15min":0.22600000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":192.50899999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55200000000000005,"15min":0.53200000000000003},"min":{"1min":0.32500000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.76700000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.44500000000000001}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.501,"15min":0.64500000000000002},"min":{"1min":0.35399999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.624,"5min":0.79500000000000004,"15min":5.0419999999999998},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.51400000000000001,"15min":0.73199999999999998},"min":{"1min":0.40100000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":0.77700000000000002,"15min":7.2930000000000001},"last":0.42699999999999999}]},{"osd":4,"last update":"Sun Sep 15 04:04:52 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.36299999999999999,"15min":0.35499999999999998},"min":{"1min":0.223,"5min":0.222,"15min":0.222},"max":{"1min":0.46000000000000002,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.41299999999999998,"15min":0.38700000000000001},"min":{"1min":0.34499999999999997,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67300000000000004,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.37}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.39200000000000002},"min":{"1min":0.27100000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.625,"5min":0.67800000000000005,"15min":0.69999999999999996},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.39400000000000002,"15min":0.40400000000000003},"min":{"1min":0.30399999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.5,"5min":0.625,"15min":0.625},"last":0.32200000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640340,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799916,"kb_used_data":92879700,"kb_used_omap":9708,"kb_used_meta":0,"kb_avail":92879700,"statfs":{"total":95927926784,"available":95108812800,"internally_reserved":0,"allocated":95108812800,"data_stored":95108812800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9941461,"internal_metadata":0},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.316,"15min":0.34000000000000002},"min":{"1min":0.20699999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.47699999999999998,"5min":0.502,"15min":0.57699999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.33000000000000002,"15min":0.36399999999999999},"min":{"1min":0.223,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.44500000000000001,"5min":0.56999999999999995,"15min":0.65100000000000002},"last":0.40799999999999997}]},{"osd":1,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.34599999999999997},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.48999999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.35899999999999999,"15min":0.372},"min":{"1min":0.28499999999999998,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54400000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.45000000000000001}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53100000000000003,"15min":0.54100000000000004},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.67100000000000004,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.55300000000000005},"min":{"1min":0.33800000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.72299999999999998,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.71599999999999997}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.66200000000000003,"5min":0.745,"15min":0.745},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.54100000000000004,"15min":0.63500000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.754,"5min":1.208,"15min":3.1339999999999999},"last":0.66900000000000004}]},{"osd":5,"last update":"Sun Sep 15 04:04:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.35999999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54000000000000004,"15min":0.65400000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73199999999999998,"5min":0.77800000000000002,"15min":2.9500000000000002},"last":0.625}]}]},{"osd":1,"up_from":11,"seq":47244640340,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":799920,"kb_used_data":92879696,"kb_used_omap":9682,"kb_used_meta":0,"kb_avail":92879696,"statfs":{"total":95927926784,"available":95108808704,"internally_reserved":0,"allocated":95108808704,"data_stored":95108808704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9914432,"internal_metadata":0},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34499999999999997,"15min":0.34300000000000003},"min":{"1min":0.222,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.52700000000000002,"5min":1.101,"15min":1.101},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.35299999999999998,"15min":0.35099999999999998},"min":{"1min":0.26400000000000001,"5min":0.182,"15min":0.182},"max":{"1min":0.497,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.34000000000000002}]},{"osd":2,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.376,"15min":0.36699999999999999},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":0.55600000000000005,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.374,"5min":0.36199999999999999,"15min":0.374},"min":{"1min":0.26300000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.5,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.378}]},{"osd":3,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52200000000000002,"15min":0.54800000000000004},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.247},"max":{"1min":0.65400000000000003,"5min":1.087,"15min":1.087},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.53600000000000003,"15min":0.54900000000000004},"min":{"1min":0.35999999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68500000000000005,"5min":1.032,"15min":1.032},"last":0.51300000000000001}]},{"osd":4,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.57699999999999996,"15min":0.56100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66400000000000003,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.623,"15min":0.59399999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68300000000000005,"5min":4.2930000000000001,"15min":4.2930000000000001},"last":0.55500000000000005}]},{"osd":5,"last update":"Sun Sep 15 04:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.33000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.72899999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.36899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70899999999999996,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.56599999999999995}]}]},{"osd":0,"up_from":11,"seq":47244640340,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93679616,"kb_used":800492,"kb_used_data":92879124,"kb_used_omap":10037,"kb_used_meta":0,"kb_avail":92879124,"statfs":{"total":95927926784,"available":95108222976,"internally_reserved":0,"allocated":95108222976,"data_stored":95108222976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10278348,"internal_metadata":0},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36099999999999999,"15min":0.34499999999999997},"min":{"1min":0.246,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.49099999999999999,"5min":3.4849999999999999,"15min":3.4849999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.35999999999999999},"min":{"1min":0.309,"5min":0.193,"15min":0.193},"max":{"1min":0.496,"5min":0.504,"15min":0.71199999999999997},"last":0.28999999999999998}]},{"osd":2,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.34799999999999998,"15min":0.376},"min":{"1min":0.32400000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52000000000000002,"5min":0.67500000000000004,"15min":0.78300000000000003},"last":0.309},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.38800000000000001},"min":{"1min":0.29199999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.54100000000000004,"5min":0.65200000000000002,"15min":0.76000000000000001},"last":0.33100000000000002}]},{"osd":3,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.43099999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.80100000000000005,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.54900000000000004,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.72299999999999998,"5min":0.85599999999999998,"15min":0.88200000000000001},"last":0.45400000000000001}]},{"osd":4,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.46300000000000002,"5min":0.30099999999999999,"15min":0.29599999999999999},"max":{"1min":0.84599999999999997,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.57199999999999995,"15min":0.59599999999999997},"min":{"1min":0.53100000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.55900000000000005}]},{"osd":5,"last update":"Sun Sep 15 04:04:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.52800000000000002,"15min":0.54800000000000004},"min":{"1min":0.44500000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73999999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.56200000000000006,"15min":0.54800000000000004},"min":{"1min":0.47399999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76000000000000001,"5min":1.044,"15min":1.044},"last":0.495}]}]}],"pool_statfs":[]}} 2024-09-15T04:05:31.153 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-15T04:05:31.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.244+0000 7f96ab834640 10 monclient: get_monmap_and_config 2024-09-15T04:05:31.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.244+0000 7f96ab834640 10 monclient: build_initial_monmap 2024-09-15T04:05:31.245 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.244+0000 7f96ab834640 10 monclient: monmap: 2024-09-15T04:05:31.245 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:31.245 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:31.245 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T04:05:31.245558+0000 2024-09-15T04:05:31.245 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T04:05:31.245558+0000 2024-09-15T04:05:31.245 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:31.246 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:31.246 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.110:6789/0 mon.noname-a-legacy 2024-09-15T04:05:31.246 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.177:6789/0 mon.noname-c-legacy 2024-09-15T04:05:31.246 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-b 2024-09-15T04:05:31.246 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.110:3300/0 mon.noname-a 2024-09-15T04:05:31.246 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.177:3300/0 mon.noname-c 2024-09-15T04:05:31.246 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:31.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 1 Processor -- start 2024-09-15T04:05:31.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 1 -- start start 2024-09-15T04:05:31.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 10 monclient: init 2024-09-15T04:05:31.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:31.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-15T04:05:31.248 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96a4131630 addr v1:172.21.15.110:6789/0 2024-09-15T04:05:31.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 1 --2- >> v2:172.21.15.177:3300/0 conn(0x7f96a406e5a0 0x7f96a406e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:31.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 10 monclient(hunting): picked mon.noname-c con 0x7f96a406e5a0 addr v2:172.21.15.177:3300/0 2024-09-15T04:05:31.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f96a406eeb0 0x7f96a4138970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:31.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 10 monclient(hunting): picked mon.noname-a con 0x7f96a406eeb0 addr v2:172.21.15.110:3300/0 2024-09-15T04:05:31.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a4068390 con 0x7f96a4131630 2024-09-15T04:05:31.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:31.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96a4069570 con 0x7f96a406eeb0 2024-09-15T04:05:31.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:31.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 1 -- --> v2:172.21.15.177:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96a4069850 con 0x7f96a406e5a0 2024-09-15T04:05:31.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:31.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96ab834640 10 monclient(hunting): authenticate will time out at 1154.889860s 2024-09-15T04:05:31.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96a9dd1640 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f96a406eeb0 0x7f96a4138970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:31.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96a95d0640 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f96a4131630 0x7f96a4131a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:36824/0 (socket says 172.21.15.110:36824) 2024-09-15T04:05:31.249 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96a95d0640 1 -- 172.21.15.110:0/887162665 learned_addr learned my addr 172.21.15.110:0/887162665 (peer_addr_for_me v1:172.21.15.110:0/0) 2024-09-15T04:05:31.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.245+0000 7f96a9dd1640 10 monclient(hunting): get_auth_request con 0x7f96a406eeb0 auth_method 0 2024-09-15T04:05:31.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:31.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:31.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:31.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:31.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:31.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:31.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 10 monclient(hunting): handle_auth_done global_id 55275 payload 274 2024-09-15T04:05:31.250 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a8dcf640 1 --2- 172.21.15.110:0/887162665 >> v2:172.21.15.177:3300/0 conn(0x7f96a406e5a0 0x7f96a406e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:31.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 1 -- 172.21.15.110:0/887162665 >> v2:172.21.15.177:3300/0 conn(0x7f96a406e5a0 msgr2=0x7f96a406e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:31.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 1 --2- 172.21.15.110:0/887162665 >> v2:172.21.15.177:3300/0 conn(0x7f96a406e5a0 0x7f96a406e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:31.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 1 -- 172.21.15.110:0/887162665 >> v1:172.21.15.110:6789/0 conn(0x7f96a4131630 legacy=0x7f96a4131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T04:05:31.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 10 monclient: _finish_hunting 0 2024-09-15T04:05:31.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 1 monclient: found mon.noname-a 2024-09-15T04:05:31.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:31.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.110:3300/0 2024-09-15T04:05:31.251 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.246+0000 7f96a9dd1640 1 -- 172.21.15.110:0/887162665 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a4132110 con 0x7f96a406eeb0 2024-09-15T04:05:31.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f96a9dd1640 1 -- 172.21.15.110:0/887162665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96a406eeb0 msgr2=0x7f96a4138970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:31.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f96a9dd1640 1 --2- 172.21.15.110:0/887162665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96a406eeb0 0x7f96a4138970 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f96a4069d10 tx=0x7f969c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b5c4f9ca56cb81a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:31.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f969bfff640 1 -- 172.21.15.110:0/887162665 <== mon.0 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c007840 con 0x7f96a406eeb0 2024-09-15T04:05:31.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f969bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:31.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f969bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T04:05:31.260 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f969bfff640 10 monclient: dump: 2024-09-15T04:05:31.260 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:31.260 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:31.260 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:31.261 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:31.264 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:31.264 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:31.264 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:31.265 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:31.265 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:31.265 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:31.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f969bfff640 10 monclient: _finish_auth 0 2024-09-15T04:05:31.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f969bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:31.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f969bfff640 1 -- 172.21.15.110:0/887162665 <== mon.0 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969c02d050 con 0x7f96a406eeb0 2024-09-15T04:05:31.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f96ab834640 5 monclient: authenticate success, global_id 55275 2024-09-15T04:05:31.265 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f96ab834640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T04:05:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f969bfff640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f96ab834640 10 monclient: get_monmap_and_config success 2024-09-15T04:05:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f969bfff640 1 -- 172.21.15.110:0/887162665 <== mon.0 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c007bd0 con 0x7f96a406eeb0 2024-09-15T04:05:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f969bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f96ab834640 4 set_mon_vals no callback set 2024-09-15T04:05:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f969bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T04:05:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f969bfff640 10 monclient: dump: 2024-09-15T04:05:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:31.266 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:31.267 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:31.267 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:31.267 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:31.267 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:31.267 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:31.267 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f96ab834640 10 monclient: shutdown 2024-09-15T04:05:31.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f96ab834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:31.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f96ab834640 1 -- 172.21.15.110:0/887162665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96a406eeb0 msgr2=0x7f96a4138970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:31.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f96ab834640 1 --2- 172.21.15.110:0/887162665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96a406eeb0 0x7f96a4138970 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f96a4069d10 tx=0x7f969c009780 comp rx=0 tx=0).stop 2024-09-15T04:05:31.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f96ab834640 1 -- 172.21.15.110:0/887162665 shutdown_connections 2024-09-15T04:05:31.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f96ab834640 1 --2- 172.21.15.110:0/887162665 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96a406eeb0 0x7f96a4138970 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:31.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f96ab834640 1 --2- 172.21.15.110:0/887162665 >> v2:172.21.15.177:3300/0 conn(0x7f96a406e5a0 0x7f96a406e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:31.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.249+0000 7f96ab834640 1 -- 172.21.15.110:0/887162665 >> 172.21.15.110:0/887162665 conn(0x7f96a40639f0 msgr2=0x7f96a40a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:31.268 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.250+0000 7f96ab834640 1 -- 172.21.15.110:0/887162665 shutdown_connections 2024-09-15T04:05:31.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.250+0000 7f96ab834640 1 -- 172.21.15.110:0/887162665 wait complete. 2024-09-15T04:05:31.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.250+0000 7f96ab834640 10 monclient: build_initial_monmap 2024-09-15T04:05:31.269 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.250+0000 7f96ab834640 10 monclient: monmap: 2024-09-15T04:05:31.269 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2024-09-15T04:05:31.269 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T04:05:31.269 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2024-09-15T04:05:31.269 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2024-09-15T04:05:31.269 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2024-09-15T04:05:31.269 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:31.269 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-a 2024-09-15T04:05:31.269 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.noname-b 2024-09-15T04:05:31.270 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.noname-c 2024-09-15T04:05:31.270 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:31.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.250+0000 7f96ab834640 1 Processor -- start 2024-09-15T04:05:31.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.250+0000 7f96ab834640 1 -- start start 2024-09-15T04:05:31.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.250+0000 7f96ab834640 10 monclient: init 2024-09-15T04:05:31.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 10 monclient: _reopen_session rank -1 2024-09-15T04:05:31.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T04:05:31.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96a406e5a0 0x7f96a4130f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:31.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 10 monclient(hunting): picked mon.noname-a con 0x7f96a406e5a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-09-15T04:05:31.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96a406eeb0 0x7f96a412bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:31.270 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 10 monclient(hunting): picked mon.noname-b con 0x7f96a406eeb0 addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-09-15T04:05:31.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96a4131630 0x7f96a412c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:31.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 10 monclient(hunting): picked mon.noname-c con 0x7f96a4131630 addr [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] 2024-09-15T04:05:31.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:31.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a412f1a0 con 0x7f96a406e5a0 2024-09-15T04:05:31.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:31.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a412f960 con 0x7f96a406eeb0 2024-09-15T04:05:31.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 10 monclient(hunting): start opening mon connection 2024-09-15T04:05:31.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 1 -- --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a412f3a0 con 0x7f96a4131630 2024-09-15T04:05:31.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 10 monclient(hunting): _renew_subs 2024-09-15T04:05:31.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96ab834640 10 monclient(hunting): authenticate will time out at 1154.895441s 2024-09-15T04:05:31.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a95d0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96a406e5a0 0x7f96a4130f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:31.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a95d0640 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96a406e5a0 0x7f96a4130f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:44126/0 (socket says 172.21.15.110:44126) 2024-09-15T04:05:31.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 1 --2- >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96a4131630 0x7f96a412c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:31.271 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a95d0640 1 -- 172.21.15.110:0/1606864989 learned_addr learned my addr 172.21.15.110:0/1606864989 (peer_addr_for_me v2:172.21.15.110:0/0) 2024-09-15T04:05:31.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a95d0640 10 monclient(hunting): get_auth_request con 0x7f96a406e5a0 auth_method 0 2024-09-15T04:05:31.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a95d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:31.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a95d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:31.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a95d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:31.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 10 monclient(hunting): get_auth_request con 0x7f96a4131630 auth_method 0 2024-09-15T04:05:31.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:31.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:31.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:31.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a95d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:31.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a95d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:31.272 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a95d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:31.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-15T04:05:31.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-15T04:05:31.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-15T04:05:31.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a8dcf640 1 --2- 172.21.15.110:0/1606864989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96a406eeb0 0x7f96a412bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:31.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a8dcf640 10 monclient(hunting): get_auth_request con 0x7f96a406eeb0 auth_method 0 2024-09-15T04:05:31.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a8dcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-15T04:05:31.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a8dcf640 10 monclient(hunting): _init_auth method 2 2024-09-15T04:05:31.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a8dcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T04:05:31.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 10 monclient(hunting): handle_auth_done global_id 74228 payload 824 2024-09-15T04:05:31.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 1 -- 172.21.15.110:0/1606864989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96a406eeb0 msgr2=0x7f96a412bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:31.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 1 --2- 172.21.15.110:0/1606864989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96a406eeb0 0x7f96a412bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:31.273 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 1 -- 172.21.15.110:0/1606864989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96a406e5a0 msgr2=0x7f96a4130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:31.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 1 --2- 172.21.15.110:0/1606864989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96a406e5a0 0x7f96a4130f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:31.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 10 monclient: _finish_hunting 0 2024-09-15T04:05:31.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 1 monclient: found mon.noname-c 2024-09-15T04:05:31.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T04:05:31.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3301/0 2024-09-15T04:05:31.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a9dd1640 1 -- 172.21.15.110:0/1606864989 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c009a20 con 0x7f96a4131630 2024-09-15T04:05:31.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.251+0000 7f96a95d0640 1 --2- 172.21.15.110:0/1606864989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96a406e5a0 0x7f96a4130f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T04:05:31.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96a9dd1640 1 -- 172.21.15.110:0/1606864989 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96a4131630 msgr2=0x7f96a412c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T04:05:31.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96a9dd1640 1 --2- 172.21.15.110:0/1606864989 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96a4131630 0x7f96a412c130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f969c031000 tx=0x7f969c037000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:31.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96997fa640 1 -- 172.21.15.110:0/1606864989 <== mon.2 v2:172.21.15.110:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c031da0 con 0x7f96a4131630 2024-09-15T04:05:31.274 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96997fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T04:05:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96997fa640 10 monclient: dump: 2024-09-15T04:05:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:31.275 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96997fa640 10 monclient: _finish_auth 0 2024-09-15T04:05:31.275 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T04:05:31.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96ab834640 5 monclient: authenticate success, global_id 74228 2024-09-15T04:05:31.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96997fa640 1 -- 172.21.15.110:0/1606864989 <== mon.2 v2:172.21.15.110:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969c040070 con 0x7f96a4131630 2024-09-15T04:05:31.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96ab834640 10 monclient: _renew_subs 2024-09-15T04:05:31.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96ab834640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:31.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96ab834640 1 -- 172.21.15.110:0/1606864989 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a4130500 con 0x7f96a4131630 2024-09-15T04:05:31.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96997fa640 10 monclient: handle_config config(0 keys) v1 2024-09-15T04:05:31.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96997fa640 1 -- 172.21.15.110:0/1606864989 <== mon.2 v2:172.21.15.110:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c045680 con 0x7f96a4131630 2024-09-15T04:05:31.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T04:05:31.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T04:05:31.276 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96997fa640 10 monclient: dump: 2024-09-15T04:05:31.276 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2024-09-15T04:05:31.276 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 802f2aac-d77f-438d-80e9-5e0c23519086 2024-09-15T04:05:31.276 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:created 2024-09-15T03:58:15.825657+0000 2024-09-15T04:05:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2024-09-15T04:05:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2024-09-15T04:05:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.a 2024-09-15T04:05:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.b 2024-09-15T04:05:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] mon.c 2024-09-15T04:05:31.277 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f969b7fe640 4 set_mon_vals no callback set 2024-09-15T04:05:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96ab834640 10 monclient: _renew_subs 2024-09-15T04:05:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96ab834640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.254+0000 7f96ab834640 1 -- 172.21.15.110:0/1606864989 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a412fd80 con 0x7f96a4131630 2024-09-15T04:05:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.255+0000 7f96997fa640 1 -- 172.21.15.110:0/1606864989 <== mon.2 v2:172.21.15.110:3301/0 4 ==== mgrmap(e 5) v1 ==== 44975+0+0 (secure 0 0 0) 0x7f969c042030 con 0x7f96a4131630 2024-09-15T04:05:31.277 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.255+0000 7f96ab834640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:31.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.255+0000 7f96ab834640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:31.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.255+0000 7f96ab834640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:31.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.255+0000 7f96ab834640 1 -- 172.21.15.110:0/1606864989 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96a4070950 con 0x7f96a4131630 2024-09-15T04:05:31.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.255+0000 7f96997fa640 1 --2- 172.21.15.110:0/1606864989 >> v2:172.21.15.177:6812/28201 conn(0x7f9688038700 0x7f968803abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T04:05:31.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.255+0000 7f96997fa640 1 -- 172.21.15.110:0/1606864989 <== mon.2 v2:172.21.15.110:3301/0 5 ==== osd_map(315..315 src has 1..315) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f969c06d120 con 0x7f96a4131630 2024-09-15T04:05:31.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.255+0000 7f96a95d0640 1 --2- 172.21.15.110:0/1606864989 >> v2:172.21.15.177:6812/28201 conn(0x7f9688038700 0x7f968803abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T04:05:31.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.255+0000 7f96a95d0640 10 monclient: get_auth_request con 0x7f9688038700 auth_method 0 2024-09-15T04:05:31.278 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.258+0000 7f96a95d0640 1 --2- 172.21.15.110:0/1606864989 >> v2:172.21.15.177:6812/28201 conn(0x7f9688038700 0x7f968803abb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f96a0006fd0 tx=0x7f96a0004960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T04:05:31.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.259+0000 7f96997fa640 1 -- 172.21.15.110:0/1606864989 <== mon.2 v2:172.21.15.110:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175657 (secure 0 0 0) 0x7f969c03c420 con 0x7f96a4131630 2024-09-15T04:05:31.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.259+0000 7f96997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T04:05:31.279 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.259+0000 7f96997fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T04:05:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.413+0000 7f96ab834640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T04:05:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.413+0000 7f96ab834640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T04:05:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.413+0000 7f96ab834640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.110:3301/0 2024-09-15T04:05:31.414 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.413+0000 7f96ab834640 1 -- 172.21.15.110:0/1606864989 --> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f96a4138c50 con 0x7f96a4131630 2024-09-15T04:05:31.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.417+0000 7f96997fa640 1 -- 172.21.15.110:0/1606864989 <== mon.2 v2:172.21.15.110:3301/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f969c03c420 con 0x7f96a4131630 2024-09-15T04:05:31.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.417+0000 7f96997fa640 10 monclient: handle_config config(1 keys) v1 2024-09-15T04:05:31.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.417+0000 7f969b7fe640 4 set_mon_vals no callback set 2024-09-15T04:05:31.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.417+0000 7f969b7fe640 10 set_mon_vals mon_health_to_clog = false 2024-09-15T04:05:31.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.417+0000 7f96997fa640 1 -- 172.21.15.110:0/1606864989 <== mon.2 v2:172.21.15.110:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f969c0303a0 con 0x7f96a4131630 2024-09-15T04:05:31.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.417+0000 7f96997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T04:05:31.418 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.417+0000 7f96997fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T04:05:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.419+0000 7f96ab834640 1 -- 172.21.15.110:0/1606864989 >> v2:172.21.15.177:6812/28201 conn(0x7f9688038700 msgr2=0x7f968803abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.419+0000 7f96ab834640 1 --2- 172.21.15.110:0/1606864989 >> v2:172.21.15.177:6812/28201 conn(0x7f9688038700 0x7f968803abb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f96a0006fd0 tx=0x7f96a0004960 comp rx=0 tx=0).stop 2024-09-15T04:05:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.419+0000 7f96ab834640 10 monclient: shutdown 2024-09-15T04:05:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.419+0000 7f96ab834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T04:05:31.420 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.419+0000 7f96ab834640 1 -- 172.21.15.110:0/1606864989 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96a4131630 msgr2=0x7f96a412c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T04:05:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.419+0000 7f96ab834640 1 --2- 172.21.15.110:0/1606864989 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96a4131630 0x7f96a412c130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f969c031000 tx=0x7f969c037000 comp rx=0 tx=0).stop 2024-09-15T04:05:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.419+0000 7f96ab834640 1 -- 172.21.15.110:0/1606864989 shutdown_connections 2024-09-15T04:05:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.419+0000 7f96ab834640 1 --2- 172.21.15.110:0/1606864989 >> v2:172.21.15.177:6812/28201 conn(0x7f9688038700 0x7f968803abb0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.419+0000 7f96ab834640 1 --2- 172.21.15.110:0/1606864989 >> [v2:172.21.15.110:3301/0,v1:172.21.15.110:6790/0] conn(0x7f96a4131630 0x7f96a412c130 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.419+0000 7f96ab834640 1 --2- 172.21.15.110:0/1606864989 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96a406eeb0 0x7f96a412bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.419+0000 7f96ab834640 1 --2- 172.21.15.110:0/1606864989 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f96a406e5a0 0x7f96a4130f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T04:05:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.419+0000 7f96ab834640 1 -- 172.21.15.110:0/1606864989 >> 172.21.15.110:0/1606864989 conn(0x7f96a40639f0 msgr2=0x7f96a412a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T04:05:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.419+0000 7f96ab834640 1 -- 172.21.15.110:0/1606864989 shutdown_connections 2024-09-15T04:05:31.421 INFO:teuthology.orchestra.run.smithi110.stderr:2024-09-15T04:05:31.419+0000 7f96ab834640 1 -- 172.21.15.110:0/1606864989 wait complete. 2024-09-15T04:05:31.430 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-15T04:05:31.431 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-15T04:05:31.431 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-15T04:05:31.431 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T04:05:31.448 INFO:tasks.ceph.osd.0:Stopped 2024-09-15T04:05:31.448 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T04:05:31.448 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T04:05:31.501 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T04:05:31.501 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-15T04:05:31.501 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T04:05:31.553 INFO:tasks.ceph.osd.2:Stopped 2024-09-15T04:05:31.554 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T04:05:31.554 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T04:05:31.570 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T04:05:31.570 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-15T04:05:31.570 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T04:05:31.626 INFO:tasks.ceph.osd.4:Stopped 2024-09-15T04:05:31.626 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-15T04:05:31.626 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T04:05:31.681 INFO:tasks.ceph.osd.5:Stopped 2024-09-15T04:05:31.681 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-15T04:05:31.681 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-15T04:05:31.682 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T04:05:31.749 INFO:tasks.ceph.mgr.x:Stopped 2024-09-15T04:05:31.749 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-15T04:05:31.749 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T04:05:31.750 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T04:05:31.761 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T04:05:31.762 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T04:05:31.762 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T04:05:31.812 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T04:05:31.812 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T04:05:31.812 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T04:05:31.826 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T04:05:31.827 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-15T04:05:31.827 DEBUG:teuthology.orchestra.run.smithi110:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | head -n 1 2024-09-15T04:05:31.881 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi110.front.sepia.ceph.com 2024-09-15T04:05:31.882 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-15T04:05:32.111 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi110.front.sepia.ceph.com 2024-09-15T04:05:32.111 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-15T04:05:32.203 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi110.front.sepia.ceph.com 2024-09-15T04:05:32.203 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-15T04:05:32.268 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-15T04:05:32.269 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-15T04:05:32.423 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-15T04:05:32.424 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-15T04:05:32.504 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-15T04:05:32.504 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-15T04:05:32.559 INFO:tasks.ceph:Archiving mon data... 2024-09-15T04:05:32.560 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905333/data/mon.a.tgz 2024-09-15T04:05:32.561 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-09-15T04:05:32.577 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.s4bs2ORhpD 2024-09-15T04:05:32.577 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.s4bs2ORhpD 2024-09-15T04:05:32.699 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.s4bs2ORhpD 2024-09-15T04:05:32.784 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.s4bs2ORhpD is 163KB 2024-09-15T04:05:32.839 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.s4bs2ORhpD 2024-09-15T04:05:32.854 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905333/data/mon.c.tgz 2024-09-15T04:05:32.854 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-09-15T04:05:32.908 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.J3dVBYduMj 2024-09-15T04:05:32.909 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.J3dVBYduMj 2024-09-15T04:05:33.001 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.J3dVBYduMj 2024-09-15T04:05:33.084 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.J3dVBYduMj is 160KB 2024-09-15T04:05:33.137 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.J3dVBYduMj 2024-09-15T04:05:33.152 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905333/data/mon.b.tgz 2024-09-15T04:05:33.153 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-09-15T04:05:33.171 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.Jcmgr4kYL2 2024-09-15T04:05:33.172 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Jcmgr4kYL2 2024-09-15T04:05:33.284 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.Jcmgr4kYL2 2024-09-15T04:05:33.372 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.Jcmgr4kYL2 is 166KB 2024-09-15T04:05:33.425 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.Jcmgr4kYL2 2024-09-15T04:05:33.441 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-15T04:05:33.441 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T04:05:33.443 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T04:05:33.505 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-15T04:05:33.506 DEBUG:teuthology.orchestra.run.smithi110:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T04:05:33.508 DEBUG:teuthology.orchestra.run.smithi177:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T04:05:33.532 INFO:teuthology.orchestra.run.smithi110.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T04:05:33.573 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T04:05:33.577 INFO:tasks.ceph:Archiving crash dumps... 2024-09-15T04:05:33.579 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905333/remote/smithi110/crash 2024-09-15T04:05:33.580 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T04:05:33.605 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905333/remote/smithi177/crash 2024-09-15T04:05:33.606 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T04:05:33.642 INFO:tasks.ceph:Compressing logs... 2024-09-15T04:05:33.643 DEBUG:teuthology.orchestra.run.smithi110:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T04:05:33.650 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T04:05:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-15T04:05:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33025.log 2024-09-15T04:05:33.672 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31026.log 2024-09-15T04:05:33.741 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.33025.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26970.log 2024-09-15T04:05:33.741 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33109.log 2024-09-15T04:05:33.741 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33025.log.gz 2024-09-15T04:05:33.741 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30541.log 2024-09-15T04:05:33.741 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.tmp-client.admin.26970.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26970.log.gz 2024-09-15T04:05:33.741 INFO:teuthology.orchestra.run.smithi110.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.31026.log.gz 2024-09-15T04:05:33.741 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.33109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28850.log 2024-09-15T04:05:33.741 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30645.log 2024-09-15T04:05:33.741 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33109.log.gz 2024-09-15T04:05:33.742 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.28850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32983.log 2024-09-15T04:05:33.742 INFO:teuthology.orchestra.run.smithi110.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30541.log.gz 2024-09-15T04:05:33.742 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29551.log 2024-09-15T04:05:33.742 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30645.log: /var/log/ceph/ceph-client.admin.32983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29126.log 2024-09-15T04:05:33.742 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28850.log.gz 2024-09-15T04:05:33.742 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29551.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30645.log.gz 2024-09-15T04:05:33.742 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32773.log 2024-09-15T04:05:33.742 INFO:teuthology.orchestra.run.smithi110.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32983.log.gz 2024-09-15T04:05:33.742 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29951.log 2024-09-15T04:05:33.742 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29551.log.gz 2024-09-15T04:05:33.742 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29126.log: /var/log/ceph/ceph-client.admin.32773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28137.log 2024-09-15T04:05:33.742 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29125.log 2024-09-15T04:05:33.742 INFO:teuthology.orchestra.run.smithi110.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29126.log.gz 2024-09-15T04:05:33.743 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32773.log.gz 2024-09-15T04:05:33.743 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.28137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29867.log 2024-09-15T04:05:33.743 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29951.log.gz 2024-09-15T04:05:33.743 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29378.log 2024-09-15T04:05:33.743 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29867.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.28137.log.gz 2024-09-15T04:05:33.743 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31640.log 2024-09-15T04:05:33.743 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29125.log.gz 2024-09-15T04:05:33.743 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29867.log.gz 2024-09-15T04:05:33.743 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32941.log 2024-09-15T04:05:33.743 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31640.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-15T04:05:33.743 INFO:teuthology.orchestra.run.smithi110.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29378.log.gz 2024-09-15T04:05:33.743 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31853.log 2024-09-15T04:05:33.743 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31640.log.gz 2024-09-15T04:05:33.744 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-osd.1.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32941.log.gz 2024-09-15T04:05:33.744 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31595.log 2024-09-15T04:05:33.744 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32107.log 2024-09-15T04:05:33.744 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31811.log 2024-09-15T04:05:33.744 INFO:teuthology.orchestra.run.smithi110.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31853.log.gz 2024-09-15T04:05:33.744 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28985.log 2024-09-15T04:05:33.744 INFO:teuthology.orchestra.run.smithi110.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31595.log.gz 2024-09-15T04:05:33.744 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30583.log 2024-09-15T04:05:33.744 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.28985.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32107.log.gz 2024-09-15T04:05:33.744 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31510.log 2024-09-15T04:05:33.744 INFO:teuthology.orchestra.run.smithi110.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31811.log.gz 2024-09-15T04:05:33.744 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29124.log 2024-09-15T04:05:33.744 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28985.log.gz 2024-09-15T04:05:33.745 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30583.log: /var/log/ceph/ceph-client.admin.31510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30899.log 2024-09-15T04:05:33.745 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30373.log 2024-09-15T04:05:33.745 INFO:teuthology.orchestra.run.smithi110.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30583.log.gz 2024-09-15T04:05:33.745 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31510.log.gz 2024-09-15T04:05:33.745 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30899.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29124.log.gz 2024-09-15T04:05:33.745 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32689.log 2024-09-15T04:05:33.745 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29127.log 2024-09-15T04:05:33.745 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30373.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30899.log.gz 2024-09-15T04:05:33.745 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29374.log 2024-09-15T04:05:33.745 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30373.log.gz 2024-09-15T04:05:33.745 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30077.log 2024-09-15T04:05:33.745 INFO:teuthology.orchestra.run.smithi110.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32689.log.gz 2024-09-15T04:05:33.745 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29374.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29127.log.gz 2024-09-15T04:05:33.746 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28943.log 2024-09-15T04:05:33.746 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32857.log 2024-09-15T04:05:33.746 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30077.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29374.log.gz 2024-09-15T04:05:33.746 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.28943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29121.log 2024-09-15T04:05:33.746 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32857.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30331.log 2024-09-15T04:05:33.746 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30077.log.gz 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28943.log.gz 2024-09-15T04:05:33.746 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:33.746 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32857.log.gz 2024-09-15T04:05:33.746 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31895.log 2024-09-15T04:05:33.746 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30331.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T04:05:33.746 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29121.log.gz 2024-09-15T04:05:33.746 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33277.log 2024-09-15T04:05:33.746 INFO:teuthology.orchestra.run.smithi110.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30331.log.gz 2024-09-15T04:05:33.746 INFO:teuthology.orchestra.run.smithi110.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31895.log.gz 2024-09-15T04:05:33.747 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33317.log 2024-09-15T04:05:33.747 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.33277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29783.log 2024-09-15T04:05:33.747 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.33317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29667.log 2024-09-15T04:05:33.747 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33277.log.gz 2024-09-15T04:05:33.747 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29783.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33317.log.gz 2024-09-15T04:05:33.747 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31979.log 2024-09-15T04:05:33.747 INFO:teuthology.orchestra.run.smithi110.stderr: 92.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T04:05:33.747 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29667.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29783.log.gz 2024-09-15T04:05:33.747 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32023.log 2024-09-15T04:05:33.747 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32311.log 2024-09-15T04:05:33.747 INFO:teuthology.orchestra.run.smithi110.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29667.log.gz 2024-09-15T04:05:33.747 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29379.log 2024-09-15T04:05:33.747 INFO:teuthology.orchestra.run.smithi110.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31979.log.gz 2024-09-15T04:05:33.747 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28769.log 2024-09-15T04:05:33.748 INFO:teuthology.orchestra.run.smithi110.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.32023.log.gz 2024-09-15T04:05:33.748 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30289.log 2024-09-15T04:05:33.748 INFO:teuthology.orchestra.run.smithi110.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.32311.log.gz 2024-09-15T04:05:33.748 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29122.log 2024-09-15T04:05:33.748 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.28769.log: /var/log/ceph/ceph-client.admin.30289.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29379.log.gz 2024-09-15T04:05:33.748 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32605.log 2024-09-15T04:05:33.748 INFO:teuthology.orchestra.run.smithi110.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28769.log.gz 2024-09-15T04:05:33.748 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31152.log 2024-09-15T04:05:33.748 INFO:teuthology.orchestra.run.smithi110.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.30289.log.gz 2024-09-15T04:05:33.748 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28825.log 2024-09-15T04:05:33.748 INFO:teuthology.orchestra.run.smithi110.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29122.log.gz 2024-09-15T04:05:33.748 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29370.log 2024-09-15T04:05:33.748 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.28825.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32605.log.gz 2024-09-15T04:05:33.749 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-15T04:05:33.749 INFO:teuthology.orchestra.run.smithi110.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31152.log.gz 2024-09-15T04:05:33.749 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32395.log 2024-09-15T04:05:33.749 INFO:teuthology.orchestra.run.smithi110.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28825.log.gz 2024-09-15T04:05:33.749 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28686.log 2024-09-15T04:05:33.749 INFO:teuthology.orchestra.run.smithi110.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29370.log.gz 2024-09-15T04:05:33.749 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29803.log 2024-09-15T04:05:33.749 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.28686.log: gzip -5 --verbose -- 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32395.log.gz 2024-09-15T04:05:33.749 INFO:teuthology.orchestra.run.smithi110.stderr: /var/log/ceph/ceph-client.admin.29375.log 2024-09-15T04:05:33.749 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29803.log: gzip -5 --verbose 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28686.log.gz 2024-09-15T04:05:33.749 INFO:teuthology.orchestra.run.smithi110.stderr: -- /var/log/ceph/ceph-client.admin.32899.log 2024-09-15T04:05:33.749 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29375.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29803.log.gz 2024-09-15T04:05:33.749 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31377.log 2024-09-15T04:05:33.749 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29375.log.gz 2024-09-15T04:05:33.750 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30984.log 2024-09-15T04:05:33.750 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30815.log 2024-09-15T04:05:33.750 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31377.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32899.log.gz 2024-09-15T04:05:33.750 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30984.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-15T04:05:33.750 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30815.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30984.log.gz 2024-09-15T04:05:33.750 INFO:teuthology.orchestra.run.smithi110.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31377.log.gz 2024-09-15T04:05:33.750 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31937.log 2024-09-15T04:05:33.750 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31682.log 2024-09-15T04:05:33.750 INFO:teuthology.orchestra.run.smithi110.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30815.log.gz 2024-09-15T04:05:33.750 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29666.log 2024-09-15T04:05:33.750 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31682.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31937.log.gz 2024-09-15T04:05:33.750 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31236.log 2024-09-15T04:05:33.750 INFO:teuthology.orchestra.run.smithi110.stderr:gzip/var/log/ceph/ceph-client.admin.29666.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30119.log 2024-09-15T04:05:33.750 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31236.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.31682.log.gz 2024-09-15T04:05:33.751 INFO:teuthology.orchestra.run.smithi110.stderr:gzip 88.7% -5 --verbose -- /var/log/ceph/ceph-client.admin.29909.log 2024-09-15T04:05:33.751 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29666.log.gz 2024-09-15T04:05:33.751 INFO:teuthology.orchestra.run.smithi110.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31236.log.gz 2024-09-15T04:05:33.751 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32437.log 2024-09-15T04:05:33.751 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30119.log: /var/log/ceph/ceph-client.admin.29909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28901.log 2024-09-15T04:05:33.751 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32437.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30119.log.gz 2024-09-15T04:05:33.751 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29909.log.gz 2024-09-15T04:05:33.751 INFO:teuthology.orchestra.run.smithi110.stderr: -- /var/log/ceph/ceph-client.admin.30035.log 2024-09-15T04:05:33.751 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.28901.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32437.log.gz 2024-09-15T04:05:33.751 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-15T04:05:33.751 INFO:teuthology.orchestra.run.smithi110.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28901.log.gz 2024-09-15T04:05:33.751 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31110.log 2024-09-15T04:05:33.751 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5/var/log/ceph/ceph-osd.2.log: --verbose -- /var/log/ceph/ceph-client.admin.31335.log 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30035.log.gz 2024-09-15T04:05:33.751 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:33.752 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29550.log 2024-09-15T04:05:33.752 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31335.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31110.log.gz 2024-09-15T04:05:33.752 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31724.log 2024-09-15T04:05:33.752 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32563.log 2024-09-15T04:05:33.752 INFO:teuthology.orchestra.run.smithi110.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31335.log.gz 2024-09-15T04:05:33.752 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31724.log: /var/log/ceph/ceph-client.admin.29550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29373.log 2024-09-15T04:05:33.752 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32563.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.31724.log.gz 2024-09-15T04:05:33.752 INFO:teuthology.orchestra.run.smithi110.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29550.log.gz 2024-09-15T04:05:33.752 INFO:teuthology.orchestra.run.smithi110.stderr:gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32563.log.gz 2024-09-15T04:05:33.752 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32269.log 2024-09-15T04:05:33.752 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32731.log 2024-09-15T04:05:33.752 INFO:teuthology.orchestra.run.smithi110.stderr: 88.5%/var/log/ceph/ceph-client.admin.32269.log: -- replaced with /var/log/ceph/ceph-client.admin.29373.log.gz 2024-09-15T04:05:33.752 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30161.log 2024-09-15T04:05:33.752 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32731.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32269.log.gz 2024-09-15T04:05:33.753 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T04:05:33.753 INFO:teuthology.orchestra.run.smithi110.stderr:gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32731.log.gz 2024-09-15T04:05:33.753 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33193.log 2024-09-15T04:05:33.753 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30161.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32647.log 2024-09-15T04:05:33.753 INFO:teuthology.orchestra.run.smithi110.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30161.log.gz 2024-09-15T04:05:33.753 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.33193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30726.log 2024-09-15T04:05:33.753 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32647.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33193.log.gz 2024-09-15T04:05:33.753 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32647.log.gz 2024-09-15T04:05:33.753 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29671.log 2024-09-15T04:05:33.753 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32479.log 2024-09-15T04:05:33.753 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29671.log: /var/log/ceph/ceph-client.admin.30726.log: 90.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T04:05:33.753 INFO:teuthology.orchestra.run.smithi110.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29671.log.gz 2024-09-15T04:05:33.753 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 91.9% -- replaced with /var/log/ceph/ceph-client.admin.30726.log.gz 2024-09-15T04:05:33.753 INFO:teuthology.orchestra.run.smithi110.stderr: --verbose -- /var/log/ceph/ceph-client.admin.31068.log 2024-09-15T04:05:33.754 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30457.log 2024-09-15T04:05:33.754 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31068.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32479.log.gz 2024-09-15T04:05:33.754 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29669.log 2024-09-15T04:05:33.754 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31068.log.gz 2024-09-15T04:05:33.754 INFO:teuthology.orchestra.run.smithi110.stderr: /var/log/ceph/ceph-client.admin.32353.log 2024-09-15T04:05:33.754 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29669.log: /var/log/ceph/ceph-client.admin.30457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32815.log 2024-09-15T04:05:33.754 INFO:teuthology.orchestra.run.smithi110.stderr: 88.6% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29669.log.gz 2024-09-15T04:05:33.754 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30457.log.gz 2024-09-15T04:05:33.754 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33235.log 2024-09-15T04:05:33.754 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32815.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32353.log.gz 2024-09-15T04:05:33.754 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30771.log 2024-09-15T04:05:33.754 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32815.log.gz 2024-09-15T04:05:33.754 INFO:teuthology.orchestra.run.smithi110.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29993.log 2024-09-15T04:05:33.754 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30771.log: /var/log/ceph/ceph-client.admin.33235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30415.log 2024-09-15T04:05:33.755 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.29993.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30771.log.gz 2024-09-15T04:05:33.755 INFO:teuthology.orchestra.run.smithi110.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33235.log.gz 2024-09-15T04:05:33.755 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29993.log.gz 2024-09-15T04:05:33.755 INFO:teuthology.orchestra.run.smithi110.stderr: /var/log/ceph/ceph-client.admin.33067.log 2024-09-15T04:05:33.755 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33151.log 2024-09-15T04:05:33.755 INFO:teuthology.orchestra.run.smithi110.stderr: 87.9%/var/log/ceph/ceph-client.admin.33067.log: -- replaced with /var/log/ceph/ceph-client.admin.30415.log.gz 2024-09-15T04:05:33.755 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31194.log 2024-09-15T04:05:33.755 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.33151.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33067.log.gz 2024-09-15T04:05:33.755 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30941.log 2024-09-15T04:05:33.755 INFO:teuthology.orchestra.run.smithi110.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30247.log 2024-09-15T04:05:33.755 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33151.log.gz 2024-09-15T04:05:33.755 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31194.log: /var/log/ceph/ceph-client.admin.30941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30857.log 2024-09-15T04:05:33.755 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30247.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30941.log.gz 2024-09-15T04:05:33.755 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31194.log.gz 2024-09-15T04:05:33.756 INFO:teuthology.orchestra.run.smithi110.stderr:gzip 89.5% -5 --verbose -- /var/log/ceph/ceph-client.admin.31293.log 2024-09-15T04:05:33.756 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30247.log.gz 2024-09-15T04:05:33.756 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30499.log 2024-09-15T04:05:33.756 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31769.log 2024-09-15T04:05:33.756 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31293.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30857.log.gz 2024-09-15T04:05:33.756 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32065.log 2024-09-15T04:05:33.756 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.31769.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.31293.log.gz 2024-09-15T04:05:33.756 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28366.log 2024-09-15T04:05:33.756 INFO:teuthology.orchestra.run.smithi110.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.31769.log.gz 2024-09-15T04:05:33.756 INFO:teuthology.orchestra.run.smithi110.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.30499.log.gz 2024-09-15T04:05:33.756 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.32065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30203.log 2024-09-15T04:05:33.756 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.28366.log: gzip 89.6% -- replaced with /var/log/ceph/ceph-client.admin.32065.log.gz 2024-09-15T04:05:33.756 INFO:teuthology.orchestra.run.smithi110.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32521.log 2024-09-15T04:05:33.756 INFO:teuthology.orchestra.run.smithi110.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28366.log.gz 2024-09-15T04:05:33.757 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/ceph-client.admin.30203.log: /var/log/ceph/ceph-client.admin.32521.log: 88.2% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30203.log.gz 2024-09-15T04:05:33.757 INFO:teuthology.orchestra.run.smithi110.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32521.log.gz 2024-09-15T04:05:33.814 INFO:teuthology.orchestra.run.smithi110.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-15T04:05:33.829 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33007.log 2024-09-15T04:05:33.829 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32799.log 2024-09-15T04:05:33.829 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5/var/log/ceph/ceph-client.admin.33007.log: --verbose -- /var/log/ceph/ceph-client.admin.34105.log 2024-09-15T04:05:33.829 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33213.log 2024-09-15T04:05:33.829 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29397.log 2024-09-15T04:05:33.829 INFO:teuthology.orchestra.run.smithi177.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.33007.log.gz 2024-09-15T04:05:33.829 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30397.log 2024-09-15T04:05:33.830 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33213.log: /var/log/ceph/ceph-client.admin.29397.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32799.log.gz 2024-09-15T04:05:33.830 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33109.log 2024-09-15T04:05:33.830 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34105.log.gz 2024-09-15T04:05:33.830 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34141.log 2024-09-15T04:05:33.830 INFO:teuthology.orchestra.run.smithi177.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.33213.log.gz 2024-09-15T04:05:33.830 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29397.log.gz 2024-09-15T04:05:33.830 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29398.log 2024-09-15T04:05:33.830 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28376.log 2024-09-15T04:05:33.830 INFO:teuthology.orchestra.run.smithi177.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33109.log.gz 2024-09-15T04:05:33.830 INFO:teuthology.orchestra.run.smithi177.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30397.log.gz 2024-09-15T04:05:33.830 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29719.log 2024-09-15T04:05:33.830 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.28376.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.34141.log.gz 2024-09-15T04:05:33.830 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33108.log 2024-09-15T04:05:33.831 INFO:teuthology.orchestra.run.smithi177.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29398.log.gz 2024-09-15T04:05:33.831 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33831.log 2024-09-15T04:05:33.831 INFO:teuthology.orchestra.run.smithi177.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28376.log.gz 2024-09-15T04:05:33.831 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29933.log 2024-09-15T04:05:33.831 INFO:teuthology.orchestra.run.smithi177.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29719.log.gz 2024-09-15T04:05:33.831 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32697.log 2024-09-15T04:05:33.831 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28456.log 2024-09-15T04:05:33.831 INFO:teuthology.orchestra.run.smithi177.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33108.log.gz 2024-09-15T04:05:33.831 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32697.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.33831.log.gz 2024-09-15T04:05:33.831 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29612.log 2024-09-15T04:05:33.831 INFO:teuthology.orchestra.run.smithi177.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29933.log.gz 2024-09-15T04:05:33.831 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.28456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33417.log 2024-09-15T04:05:33.831 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33935.log 2024-09-15T04:05:33.831 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.32697.log.gz 2024-09-15T04:05:33.832 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28456.log.gz 2024-09-15T04:05:33.832 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33384.log 2024-09-15T04:05:33.832 INFO:teuthology.orchestra.run.smithi177.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29612.log.gz 2024-09-15T04:05:33.832 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32078.log 2024-09-15T04:05:33.832 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30506.log 2024-09-15T04:05:33.832 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33417.log.gz 2024-09-15T04:05:33.832 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32903.log 2024-09-15T04:05:33.832 INFO:teuthology.orchestra.run.smithi177.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.33935.log.gz 2024-09-15T04:05:33.832 INFO:teuthology.orchestra.run.smithi177.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33384.log.gz 2024-09-15T04:05:33.832 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30943.log 2024-09-15T04:05:33.832 INFO:teuthology.orchestra.run.smithi177.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32078.log.gz 2024-09-15T04:05:33.832 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29542.log 2024-09-15T04:05:33.832 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31357.log 2024-09-15T04:05:33.833 INFO:teuthology.orchestra.run.smithi177.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.30506.log.gz 2024-09-15T04:05:33.833 INFO:teuthology.orchestra.run.smithi177.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32903.log.gz 2024-09-15T04:05:33.833 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32249.log 2024-09-15T04:05:33.833 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33178.log 2024-09-15T04:05:33.833 INFO:teuthology.orchestra.run.smithi177.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30943.log.gz 2024-09-15T04:05:33.833 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32147.log 2024-09-15T04:05:33.833 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.29542.log.gz 2024-09-15T04:05:33.833 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.31357.log.gz 2024-09-15T04:05:33.833 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33900.log 2024-09-15T04:05:33.833 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30909.log 2024-09-15T04:05:33.833 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33900.log: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.33178.log.gz 2024-09-15T04:05:33.833 INFO:teuthology.orchestra.run.smithi177.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30255.log 2024-09-15T04:05:33.833 INFO:teuthology.orchestra.run.smithi177.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32147.log.gz 2024-09-15T04:05:33.833 INFO:teuthology.orchestra.run.smithi177.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32249.log.gz 2024-09-15T04:05:33.834 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30039.log 2024-09-15T04:05:33.834 INFO:teuthology.orchestra.run.smithi177.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33900.log.gz 2024-09-15T04:05:33.834 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32388.log 2024-09-15T04:05:33.834 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30290.log 2024-09-15T04:05:33.834 INFO:teuthology.orchestra.run.smithi177.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30255.log.gz 2024-09-15T04:05:33.834 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29540.log 2024-09-15T04:05:33.834 INFO:teuthology.orchestra.run.smithi177.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30909.log.gz 2024-09-15T04:05:33.834 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33316.log 2024-09-15T04:05:33.834 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.32388.log.gz 2024-09-15T04:05:33.834 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30039.log.gz 2024-09-15T04:05:33.834 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29647.log 2024-09-15T04:05:33.834 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29113.log 2024-09-15T04:05:33.834 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33316.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30290.log.gz 2024-09-15T04:05:33.834 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.29647.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30292.log 2024-09-15T04:05:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33522.log 2024-09-15T04:05:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30292.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29540.log.gzgzip 2024-09-15T04:05:33.835 INFO:teuthology.orchestra.run.smithi177.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32902.log 2024-09-15T04:05:33.835 INFO:teuthology.orchestra.run.smithi177.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.33316.log.gz 2024-09-15T04:05:33.835 INFO:teuthology.orchestra.run.smithi177.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29113.log.gz 2024-09-15T04:05:33.835 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29647.log.gz 2024-09-15T04:05:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29504.log 2024-09-15T04:05:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29826.log 2024-09-15T04:05:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29504.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.30292.log.gz 2024-09-15T04:05:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31046.log 2024-09-15T04:05:33.835 INFO:teuthology.orchestra.run.smithi177.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.33522.log.gz 2024-09-15T04:05:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31976.log 2024-09-15T04:05:33.835 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32902.log.gz 2024-09-15T04:05:33.835 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29433.log 2024-09-15T04:05:33.836 INFO:teuthology.orchestra.run.smithi177.stderr: 88.8% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29826.log.gz 2024-09-15T04:05:33.836 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29504.log.gz 2024-09-15T04:05:33.836 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32077.log 2024-09-15T04:05:33.836 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31941.log 2024-09-15T04:05:33.836 INFO:teuthology.orchestra.run.smithi177.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31046.log.gz 2024-09-15T04:05:33.836 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32455.log 2024-09-15T04:05:33.836 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.31976.log.gz 2024-09-15T04:05:33.836 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31941.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29433.log.gz 2024-09-15T04:05:33.836 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30148.log 2024-09-15T04:05:33.836 INFO:teuthology.orchestra.run.smithi177.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31941.log.gz 2024-09-15T04:05:33.836 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32455.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32077.log.gz 2024-09-15T04:05:33.836 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28548.log 2024-09-15T04:05:33.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32695.log 2024-09-15T04:05:33.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.28548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29434.log 2024-09-15T04:05:33.837 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32455.log.gz 2024-09-15T04:05:33.837 INFO:teuthology.orchestra.run.smithi177.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30148.log.gz 2024-09-15T04:05:33.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29863.log 2024-09-15T04:05:33.837 INFO:teuthology.orchestra.run.smithi177.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28548.log.gz 2024-09-15T04:05:33.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30575.log 2024-09-15T04:05:33.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30505.log 2024-09-15T04:05:33.837 INFO:teuthology.orchestra.run.smithi177.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29434.log.gz 2024-09-15T04:05:33.837 INFO:teuthology.orchestra.run.smithi177.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32695.log.gz 2024-09-15T04:05:33.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32180.log 2024-09-15T04:05:33.837 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32490.log 2024-09-15T04:05:33.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30505.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.29863.log.gz 2024-09-15T04:05:33.837 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28508.log 2024-09-15T04:05:33.838 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30575.log.gz 2024-09-15T04:05:33.838 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29435.log 2024-09-15T04:05:33.838 INFO:teuthology.orchestra.run.smithi177.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30505.log.gz 2024-09-15T04:05:33.838 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.28508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29932.log 2024-09-15T04:05:33.838 INFO:teuthology.orchestra.run.smithi177.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32180.log.gz 2024-09-15T04:05:33.838 INFO:teuthology.orchestra.run.smithi177.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32490.log.gz 2024-09-15T04:05:33.838 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29969.log 2024-09-15T04:05:33.838 INFO:teuthology.orchestra.run.smithi177.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28508.log.gz 2024-09-15T04:05:33.838 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33589.log 2024-09-15T04:05:33.838 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33005.log 2024-09-15T04:05:33.838 INFO:teuthology.orchestra.run.smithi177.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.29435.log.gz 2024-09-15T04:05:33.838 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32559.log 2024-09-15T04:05:33.838 INFO:teuthology.orchestra.run.smithi177.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29969.log.gz 2024-09-15T04:05:33.838 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33005.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29932.log.gz 2024-09-15T04:05:33.839 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32801.log 2024-09-15T04:05:33.839 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33796.log 2024-09-15T04:05:33.839 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32559.log: gzip/var/log/ceph/ceph-client.admin.32801.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31321.log 2024-09-15T04:05:33.839 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33589.log.gz 2024-09-15T04:05:33.839 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33005.log.gz 2024-09-15T04:05:33.839 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31460.log 2024-09-15T04:05:33.839 INFO:teuthology.orchestra.run.smithi177.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32559.log.gz 2024-09-15T04:05:33.839 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33625.log 2024-09-15T04:05:33.839 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.32801.log.gz 2024-09-15T04:05:33.839 INFO:teuthology.orchestra.run.smithi177.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33796.log.gz 2024-09-15T04:05:33.839 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28416.log 2024-09-15T04:05:33.839 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31048.log 2024-09-15T04:05:33.839 INFO:teuthology.orchestra.run.smithi177.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31321.log.gz 2024-09-15T04:05:33.839 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.28416.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.31460.log.gz 2024-09-15T04:05:33.840 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31562.log 2024-09-15T04:05:33.840 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29505.log 2024-09-15T04:05:33.840 INFO:teuthology.orchestra.run.smithi177.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.33625.log.gz 2024-09-15T04:05:33.840 INFO:teuthology.orchestra.run.smithi177.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28416.log.gz 2024-09-15T04:05:33.840 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34003.log 2024-09-15T04:05:33.840 INFO:teuthology.orchestra.run.smithi177.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31048.log.gz 2024-09-15T04:05:33.840 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33486.log 2024-09-15T04:05:33.840 INFO:teuthology.orchestra.run.smithi177.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31562.log.gz 2024-09-15T04:05:33.840 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31355.log 2024-09-15T04:05:33.840 INFO:teuthology.orchestra.run.smithi177.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29505.log.gz 2024-09-15T04:05:33.840 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29291.log 2024-09-15T04:05:33.840 INFO:teuthology.orchestra.run.smithi177.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34003.log.gz 2024-09-15T04:05:33.840 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30790.log 2024-09-15T04:05:33.840 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33934.log 2024-09-15T04:05:33.841 INFO:teuthology.orchestra.run.smithi177.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33486.log.gz 2024-09-15T04:05:33.841 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31115.log 2024-09-15T04:05:33.841 INFO:teuthology.orchestra.run.smithi177.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29291.log.gz 2024-09-15T04:05:33.841 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31355.log.gz 2024-09-15T04:05:33.841 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33727.log 2024-09-15T04:05:33.841 INFO:teuthology.orchestra.run.smithi177.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30790.log.gz 2024-09-15T04:05:33.841 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30077.log 2024-09-15T04:05:33.841 INFO:teuthology.orchestra.run.smithi177.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33934.log.gz 2024-09-15T04:05:33.841 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30183.log 2024-09-15T04:05:33.841 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29327.log 2024-09-15T04:05:33.841 INFO:teuthology.orchestra.run.smithi177.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33727.log.gz 2024-09-15T04:05:33.841 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31115.log.gz 2024-09-15T04:05:33.841 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33726.log 2024-09-15T04:05:33.841 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31424.log 2024-09-15T04:05:33.842 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.30077.log.gz 2024-09-15T04:05:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31734.log 2024-09-15T04:05:33.842 INFO:teuthology.orchestra.run.smithi177.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29327.log.gz 2024-09-15T04:05:33.842 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30183.log.gz 2024-09-15T04:05:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33693.log 2024-09-15T04:05:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30612.log 2024-09-15T04:05:33.842 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33726.log.gz 2024-09-15T04:05:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34038.log 2024-09-15T04:05:33.842 INFO:teuthology.orchestra.run.smithi177.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31734.log.gz 2024-09-15T04:05:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30612.log: gzip -5 89.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.31424.log.gz 2024-09-15T04:05:33.842 INFO:teuthology.orchestra.run.smithi177.stderr: /var/log/ceph/ceph-client.admin.31047.log 2024-09-15T04:05:33.842 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33693.log.gz 2024-09-15T04:05:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29079.log 2024-09-15T04:05:33.842 INFO:teuthology.orchestra.run.smithi177.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30612.log.gz 2024-09-15T04:05:33.843 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31047.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-15T04:05:33.843 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29079.log: gzip -5 --verbose -- 90.3% /var/log/ceph/ceph-client.admin.32284.log -- replaced with /var/log/ceph/ceph-client.admin.34038.log.gz 2024-09-15T04:05:33.843 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T04:05:33.843 INFO:teuthology.orchestra.run.smithi177.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31047.log.gz 2024-09-15T04:05:33.843 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31459.log 2024-09-15T04:05:33.843 INFO:teuthology.orchestra.run.smithi177.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.29079.log.gz 2024-09-15T04:05:33.843 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32284.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T04:05:33.843 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30075.log 2024-09-15T04:05:33.843 INFO:teuthology.orchestra.run.smithi177.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32284.log.gz 2024-09-15T04:05:33.843 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31528.log 2024-09-15T04:05:33.843 INFO:teuthology.orchestra.run.smithi177.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31459.log.gz 2024-09-15T04:05:33.843 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30910.log 2024-09-15T04:05:33.843 INFO:teuthology.orchestra.run.smithi177.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T04:05:33.844 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30613.log 2024-09-15T04:05:33.844 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 88.9% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.30075.log.gz 2024-09-15T04:05:33.844 INFO:teuthology.orchestra.run.smithi177.stderr: /var/log/ceph/ceph-client.admin.33419.log 2024-09-15T04:05:33.844 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30910.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31528.log.gz 2024-09-15T04:05:33.844 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32283.log 2024-09-15T04:05:33.844 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33419.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.30910.log.gz --verbose 2024-09-15T04:05:33.844 INFO:teuthology.orchestra.run.smithi177.stderr: -- /var/log/ceph/ceph-client.admin.31940.log 2024-09-15T04:05:33.844 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32285.log 2024-09-15T04:05:33.844 INFO:teuthology.orchestra.run.smithi177.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.30613.log.gz 2024-09-15T04:05:33.844 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31356.log 2024-09-15T04:05:33.844 INFO:teuthology.orchestra.run.smithi177.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.33419.log.gz 2024-09-15T04:05:33.844 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32387.log 2024-09-15T04:05:33.844 INFO:teuthology.orchestra.run.smithi177.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32283.log.gz 2024-09-15T04:05:33.844 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31151.log 2024-09-15T04:05:33.845 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% 90.4% -- replaced with /var/log/ceph/ceph-client.admin.32285.log.gz -- replaced with /var/log/ceph/ceph-client.admin.31940.log.gz 2024-09-15T04:05:33.845 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T04:05:33.845 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.32387.log: -- /var/log/ceph/ceph-client.admin.31253.log 2024-09-15T04:05:33.845 INFO:teuthology.orchestra.run.smithi177.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31356.log.gz 2024-09-15T04:05:33.845 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32182.log 2024-09-15T04:05:33.845 INFO:teuthology.orchestra.run.smithi177.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32387.log.gz 2024-09-15T04:05:33.845 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30826.log 2024-09-15T04:05:33.845 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.32182.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30504.log 2024-09-15T04:05:33.845 INFO:teuthology.orchestra.run.smithi177.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31253.log.gz 2024-09-15T04:05:33.845 INFO:teuthology.orchestra.run.smithi177.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.31151.log.gz 2024-09-15T04:05:33.845 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33383.log 2024-09-15T04:05:33.845 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29112.log 2024-09-15T04:05:33.845 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30504.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.32182.log.gz 2024-09-15T04:05:33.845 INFO:teuthology.orchestra.run.smithi177.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.30826.log.gz 2024-09-15T04:05:33.846 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32904.log 2024-09-15T04:05:33.846 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31219.log 2024-09-15T04:05:33.846 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31425.log 2024-09-15T04:05:33.846 INFO:teuthology.orchestra.run.smithi177.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30504.log.gz 2024-09-15T04:05:33.846 INFO:teuthology.orchestra.run.smithi177.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33383.log.gz 2024-09-15T04:05:33.846 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33728.log 2024-09-15T04:05:33.846 INFO:teuthology.orchestra.run.smithi177.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29112.log.gz 2024-09-15T04:05:33.846 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29219.log 2024-09-15T04:05:33.846 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.32904.log.gz 2024-09-15T04:05:33.846 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31219.log.gz 2024-09-15T04:05:33.846 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33314.log 87.5% 2024-09-15T04:05:33.846 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31425.log.gz 2024-09-15T04:05:33.846 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33075.log 2024-09-15T04:05:33.846 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33314.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.33728.log.gz 2024-09-15T04:05:33.847 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30682.log 2024-09-15T04:05:33.847 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34139.log 2024-09-15T04:05:33.847 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29219.log.gz 2024-09-15T04:05:33.847 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32146.log 2024-09-15T04:05:33.847 INFO:teuthology.orchestra.run.smithi177.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33075.log.gz 2024-09-15T04:05:33.847 INFO:teuthology.orchestra.run.smithi177.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33314.log.gz 2024-09-15T04:05:33.847 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33624.log 2024-09-15T04:05:33.847 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31768.log 2024-09-15T04:05:33.847 INFO:teuthology.orchestra.run.smithi177.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30682.log.gz 2024-09-15T04:05:33.847 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33590.log 2024-09-15T04:05:33.847 INFO:teuthology.orchestra.run.smithi177.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34139.log.gz 2024-09-15T04:05:33.847 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29611.log 2024-09-15T04:05:33.847 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31768.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33624.log.gz 2024-09-15T04:05:33.847 INFO:teuthology.orchestra.run.smithi177.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32146.log.gz 2024-09-15T04:05:33.848 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30720.log 2024-09-15T04:05:33.848 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29754.log 2024-09-15T04:05:33.848 INFO:teuthology.orchestra.run.smithi177.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33590.log.gz 2024-09-15T04:05:33.848 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31631.log 2024-09-15T04:05:33.848 INFO:teuthology.orchestra.run.smithi177.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31768.log.gz 2024-09-15T04:05:33.848 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32869.log 2024-09-15T04:05:33.848 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29754.log: /var/log/ceph/ceph-client.admin.31631.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.30720.log.gz 2024-09-15T04:05:33.848 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33829.log 2024-09-15T04:05:33.848 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29611.log.gz 2024-09-15T04:05:33.848 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32662.log 2024-09-15T04:05:33.848 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32869.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29754.log.gz 2024-09-15T04:05:33.848 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33829.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31631.log.gz 2024-09-15T04:05:33.848 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33899.log 2024-09-15T04:05:33.848 INFO:teuthology.orchestra.run.smithi177.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32869.log.gz 2024-09-15T04:05:33.849 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33110.log 2024-09-15T04:05:33.849 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32662.log: /var/log/ceph/ceph-client.admin.33899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32558.log 2024-09-15T04:05:33.849 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33829.log.gz 2024-09-15T04:05:33.849 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33110.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32662.log.gz 2024-09-15T04:05:33.849 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32592.log 2024-09-15T04:05:33.849 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31871.log 2024-09-15T04:05:33.849 INFO:teuthology.orchestra.run.smithi177.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33899.log.gz 2024-09-15T04:05:33.849 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29756.log 2024-09-15T04:05:33.849 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.33110.log.gz 2024-09-15T04:05:33.849 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31664.log 2024-09-15T04:05:33.849 INFO:teuthology.orchestra.run.smithi177.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32558.log.gz 2024-09-15T04:05:33.849 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29862.log 2024-09-15T04:05:33.849 INFO:teuthology.orchestra.run.smithi177.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32592.log.gz 2024-09-15T04:05:33.850 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32352.log 2024-09-15T04:05:33.850 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30147.log 2024-09-15T04:05:33.850 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29862.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31871.log.gz 2024-09-15T04:05:33.850 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.29756.log.gz 2024-09-15T04:05:33.850 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31975.log 2024-09-15T04:05:33.850 INFO:teuthology.orchestra.run.smithi177.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29862.log.gz 2024-09-15T04:05:33.850 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31664.log.gz 2024-09-15T04:05:33.850 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31218.log 2024-09-15T04:05:33.850 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33795.log 2024-09-15T04:05:33.850 INFO:teuthology.orchestra.run.smithi177.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32352.log.gz 2024-09-15T04:05:33.850 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31218.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31975.log.gz 2024-09-15T04:05:33.850 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30789.log 2024-09-15T04:05:33.850 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30147.log.gz 2024-09-15T04:05:33.850 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31733.log 2024-09-15T04:05:33.851 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30611.log 2024-09-15T04:05:33.851 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31733.log: gzip -5 --verbose -- 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31218.log.gz 2024-09-15T04:05:33.851 INFO:teuthology.orchestra.run.smithi177.stderr: /var/log/ceph/ceph-client.admin.28336.log 2024-09-15T04:05:33.851 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30611.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33795.log.gz 2024-09-15T04:05:33.851 INFO:teuthology.orchestra.run.smithi177.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30789.log.gz 2024-09-15T04:05:33.851 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30718.log 2024-09-15T04:05:33.851 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31150.log 2024-09-15T04:05:33.851 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.28336.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31733.log.gz 2024-09-15T04:05:33.851 INFO:teuthology.orchestra.run.smithi177.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30611.log.gz 2024-09-15T04:05:33.851 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31837.log 2024-09-15T04:05:33.851 INFO:teuthology.orchestra.run.smithi177.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28336.log.gz 2024-09-15T04:05:33.851 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30040.log 2024-09-15T04:05:33.851 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31116.log 2024-09-15T04:05:33.851 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32594.log 2024-09-15T04:05:33.852 INFO:teuthology.orchestra.run.smithi177.stderr: 89.3% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31150.log.gz -- replaced with /var/log/ceph/ceph-client.admin.30718.log.gz 2024-09-15T04:05:33.852 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T04:05:33.852 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31563.log 2024-09-15T04:05:33.852 INFO:teuthology.orchestra.run.smithi177.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30040.log.gz 2024-09-15T04:05:33.852 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32594.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31837.log.gz 2024-09-15T04:05:33.852 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32765.log 2024-09-15T04:05:33.852 INFO:teuthology.orchestra.run.smithi177.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31116.log.gz 2024-09-15T04:05:33.852 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29290.log 2024-09-15T04:05:33.852 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.32765.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32079.log 2024-09-15T04:05:33.852 INFO:teuthology.orchestra.run.smithi177.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.32594.log.gz 2024-09-15T04:05:33.852 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29290.log: gzip 90.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.31563.log.gz -- 2024-09-15T04:05:33.852 INFO:teuthology.orchestra.run.smithi177.stderr: /var/log/ceph/ceph-client.admin.30398.log 2024-09-15T04:05:33.852 INFO:teuthology.orchestra.run.smithi177.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32765.log.gz 2024-09-15T04:05:33.852 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33933.log 2024-09-15T04:05:33.853 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32079.log: /var/log/ceph/ceph-client.admin.30398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29220.log 2024-09-15T04:05:33.853 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29718.log 2024-09-15T04:05:33.853 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29290.log.gz 2024-09-15T04:05:33.853 INFO:teuthology.orchestra.run.smithi177.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30398.log.gz 2024-09-15T04:05:33.853 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.32079.log.gz 2024-09-15T04:05:33.853 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33281.log 2024-09-15T04:05:33.853 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32661.log 2024-09-15T04:05:33.853 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29718.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29220.log.gz 2024-09-15T04:05:33.853 INFO:teuthology.orchestra.run.smithi177.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33933.log.gz 2024-09-15T04:05:33.853 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29825.log 2024-09-15T04:05:33.853 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31458.log 2024-09-15T04:05:33.853 INFO:teuthology.orchestra.run.smithi177.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33281.log.gz 2024-09-15T04:05:33.853 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31630.log 2024-09-15T04:05:33.853 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31458.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29718.log.gz 2024-09-15T04:05:33.854 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29649.log 2024-09-15T04:05:33.854 INFO:teuthology.orchestra.run.smithi177.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32661.log.gz 2024-09-15T04:05:33.854 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33418.log 2024-09-15T04:05:33.854 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29649.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29825.log.gz 2024-09-15T04:05:33.854 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29326.log 2024-09-15T04:05:33.854 INFO:teuthology.orchestra.run.smithi177.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31458.log.gz 2024-09-15T04:05:33.854 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32181.log 2024-09-15T04:05:33.854 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31630.log.gz 2024-09-15T04:05:33.854 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33487.log 2024-09-15T04:05:33.854 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29326.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29649.log.gz 2024-09-15T04:05:33.854 INFO:teuthology.orchestra.run.smithi177.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33418.log.gz 2024-09-15T04:05:33.854 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31872.log 2024-09-15T04:05:33.854 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31561.log 2024-09-15T04:05:33.854 INFO:teuthology.orchestra.run.smithi177.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32181.log.gz 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29326.log.gz 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29183.log 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31872.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33487.log.gz 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33212.log 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31872.log.gz 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33177.log 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30468.log 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33212.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31561.log.gz 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31254.log 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32868.log 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33212.log.gz 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30468.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29183.log.gz 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30362.log 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30076.log 2024-09-15T04:05:33.855 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32868.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33177.log.gz 2024-09-15T04:05:33.856 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30362.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.31254.log.gz 2024-09-15T04:05:33.856 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-15T04:05:33.856 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30468.log.gz 2024-09-15T04:05:33.856 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30576.log 2024-09-15T04:05:33.856 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30076.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30362.log.gz 2024-09-15T04:05:33.856 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T04:05:33.856 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.5.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32868.log.gz 2024-09-15T04:05:33.856 INFO:teuthology.orchestra.run.smithi177.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30076.log.gz 2024-09-15T04:05:33.856 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32696.log 2024-09-15T04:05:33.856 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33623.log 2024-09-15T04:05:33.856 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30576.log.gz 2024-09-15T04:05:33.856 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31322.log 2024-09-15T04:05:33.856 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32764.log 2024-09-15T04:05:33.857 INFO:teuthology.orchestra.run.smithi177.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32696.log.gz 2024-09-15T04:05:33.857 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32491.log 2024-09-15T04:05:33.857 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32764.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33623.log.gz 2024-09-15T04:05:33.857 INFO:teuthology.orchestra.run.smithi177.stderr: -5 --verbose 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31322.log.gz 2024-09-15T04:05:33.857 INFO:teuthology.orchestra.run.smithi177.stderr: -- /var/log/ceph/ceph-client.admin.30184.log 2024-09-15T04:05:33.857 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32043.log 2024-09-15T04:05:33.857 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33315.log 2024-09-15T04:05:33.857 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32764.log.gz/var/log/ceph/ceph-client.admin.32043.log: 2024-09-15T04:05:33.857 INFO:teuthology.orchestra.run.smithi177.stderr: 90.4% 86.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T04:05:33.857 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32491.log.gz 2024-09-15T04:05:33.857 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32456.log 2024-09-15T04:05:33.857 INFO:teuthology.orchestra.run.smithi177.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30184.log.gz 2024-09-15T04:05:33.857 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-15T04:05:33.857 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33315.log: /var/log/ceph/ceph-client.admin.32456.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32043.log.gz 2024-09-15T04:05:33.858 INFO:teuthology.orchestra.run.smithi177.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30291.log 2024-09-15T04:05:33.858 INFO:teuthology.orchestra.run.smithi177.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33315.log.gz 2024-09-15T04:05:33.858 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34037.log 2024-09-15T04:05:33.858 INFO:teuthology.orchestra.run.smithi177.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32456.log.gz 2024-09-15T04:05:33.858 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31974.log 2024-09-15T04:05:33.858 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30291.log: /var/log/ceph/ceph-client.admin.34037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31873.log 2024-09-15T04:05:33.858 INFO:teuthology.orchestra.run.smithi177.stderr: 88.6%/var/log/ceph/ceph-client.admin.31974.log: -- replaced with /var/log/ceph/ceph-client.admin.30291.log.gz 2024-09-15T04:05:33.858 INFO:teuthology.orchestra.run.smithi177.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29970.log 2024-09-15T04:05:33.858 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34037.log.gz 2024-09-15T04:05:33.858 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31666.log 2024-09-15T04:05:33.858 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31838.log 2024-09-15T04:05:33.858 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0%/var/log/ceph/ceph-client.admin.31666.log: -- replaced with /var/log/ceph/ceph-client.admin.31974.log.gz 2024-09-15T04:05:33.858 INFO:teuthology.orchestra.run.smithi177.stderr:gzip 90.3% -- replaced with /var/log/ceph/ceph-client.admin.31873.log.gz -5 2024-09-15T04:05:33.858 INFO:teuthology.orchestra.run.smithi177.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34036.log 2024-09-15T04:05:33.859 INFO:teuthology.orchestra.run.smithi177.stderr: 90.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30469.log 2024-09-15T04:05:33.859 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29970.log.gz/var/log/ceph/ceph-client.admin.31838.log: 2024-09-15T04:05:33.859 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30719.log 2024-09-15T04:05:33.859 INFO:teuthology.orchestra.run.smithi177.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.31666.log.gz 2024-09-15T04:05:33.859 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31838.log.gz 2024-09-15T04:05:33.859 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30469.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34036.log.gz 2024-09-15T04:05:33.859 INFO:teuthology.orchestra.run.smithi177.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30469.log.gz 2024-09-15T04:05:33.859 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32353.log 2024-09-15T04:05:33.859 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32250.log 2024-09-15T04:05:33.859 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29648.log 2024-09-15T04:05:33.859 INFO:teuthology.orchestra.run.smithi177.stderr: 89.5%/var/log/ceph/ceph-client.admin.32250.log: -- replaced with /var/log/ceph/ceph-client.admin.30719.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32972.log 2024-09-15T04:05:33.859 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T04:05:33.859 INFO:teuthology.orchestra.run.smithi177.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32353.log.gz 2024-09-15T04:05:33.859 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31770.log 2024-09-15T04:05:33.860 INFO:teuthology.orchestra.run.smithi177.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32250.log.gz 2024-09-15T04:05:33.860 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34002.log 2024-09-15T04:05:33.860 INFO:teuthology.orchestra.run.smithi177.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29648.log.gz 2024-09-15T04:05:33.860 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32489.log 2024-09-15T04:05:33.860 INFO:teuthology.orchestra.run.smithi177.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32972.log.gz 2024-09-15T04:05:33.860 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33692.log 2024-09-15T04:05:33.860 INFO:teuthology.orchestra.run.smithi177.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.31770.log.gz 2024-09-15T04:05:33.860 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32489.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-15T04:05:33.860 INFO:teuthology.orchestra.run.smithi177.stderr:gzip/var/log/ceph/ceph-client.admin.33692.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32593.log 2024-09-15T04:05:33.860 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mon.b.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33211.log 2024-09-15T04:05:33.860 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34002.log.gz 2024-09-15T04:05:33.860 INFO:teuthology.orchestra.run.smithi177.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32489.log.gz 2024-09-15T04:05:33.860 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29755.log 2024-09-15T04:05:33.860 INFO:teuthology.orchestra.run.smithi177.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33692.log.gz 2024-09-15T04:05:33.861 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29968.log 2024-09-15T04:05:33.861 INFO:teuthology.orchestra.run.smithi177.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32593.log.gz 2024-09-15T04:05:33.861 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33211.log: /var/log/ceph/ceph-client.admin.29755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29184.log 2024-09-15T04:05:33.861 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29968.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29755.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29221.log 2024-09-15T04:05:33.861 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T04:05:33.861 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33211.log.gz 2024-09-15T04:05:33.861 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29184.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29968.log.gz 2024-09-15T04:05:33.861 INFO:teuthology.orchestra.run.smithi177.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29184.log.gz 2024-09-15T04:05:33.861 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33280.log 2024-09-15T04:05:33.861 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29078.log 2024-09-15T04:05:33.861 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30945.log 2024-09-15T04:05:33.861 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34140.log 2024-09-15T04:05:33.861 INFO:teuthology.orchestra.run.smithi177.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29221.log.gz 2024-09-15T04:05:33.861 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30945.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33280.log.gz 2024-09-15T04:05:33.862 INFO:teuthology.orchestra.run.smithi177.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29078.log.gz 2024-09-15T04:05:33.862 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30399.log 2024-09-15T04:05:33.862 INFO:teuthology.orchestra.run.smithi177.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.30945.log.gz 2024-09-15T04:05:33.862 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33006.log 2024-09-15T04:05:33.862 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30827.log 2024-09-15T04:05:33.862 INFO:teuthology.orchestra.run.smithi177.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34140.log.gz 2024-09-15T04:05:33.862 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29114.log 2024-09-15T04:05:33.862 INFO:teuthology.orchestra.run.smithi177.stderr: 90.4%/var/log/ceph/ceph-client.admin.30827.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33006.log.gz 2024-09-15T04:05:33.862 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30399.log.gz 2024-09-15T04:05:33.862 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33074.log 2024-09-15T04:05:33.862 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32044.log 2024-09-15T04:05:33.862 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29114.log: 90.7%/var/log/ceph/ceph-client.admin.33074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30185.log 2024-09-15T04:05:33.862 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30827.log.gz 2024-09-15T04:05:33.862 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32971.log 2024-09-15T04:05:33.863 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30185.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29114.log.gz 2024-09-15T04:05:33.863 INFO:teuthology.orchestra.run.smithi177.stderr: 88.2% 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33074.log.gz 2024-09-15T04:05:33.863 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32044.log.gz 2024-09-15T04:05:33.863 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29541.log 2024-09-15T04:05:33.863 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30254.log 2024-09-15T04:05:33.863 INFO:teuthology.orchestra.run.smithi177.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.30185.log.gz/var/log/ceph/ceph-client.admin.32971.log: 2024-09-15T04:05:33.863 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30361.log 2024-09-15T04:05:33.863 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31013.log 2024-09-15T04:05:33.863 INFO:teuthology.orchestra.run.smithi177.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29541.log.gz/var/log/ceph/ceph-client.admin.30361.log: 2024-09-15T04:05:33.863 INFO:teuthology.orchestra.run.smithi177.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32971.log.gz 2024-09-15T04:05:33.863 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30254.log.gz 2024-09-15T04:05:33.863 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31149.log 2024-09-15T04:05:33.863 INFO:teuthology.orchestra.run.smithi177.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30361.log.gz 2024-09-15T04:05:33.863 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30825.log 2024-09-15T04:05:33.864 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31013.log: /var/log/ceph/ceph-client.admin.31149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32800.log 2024-09-15T04:05:33.864 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30944.log 2024-09-15T04:05:33.864 INFO:teuthology.orchestra.run.smithi177.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31013.log.gz 2024-09-15T04:05:33.864 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32800.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29328.log 2024-09-15T04:05:33.864 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31149.log.gz 2024-09-15T04:05:33.864 INFO:teuthology.orchestra.run.smithi177.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.32800.log.gz 2024-09-15T04:05:33.864 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33830.log 2024-09-15T04:05:33.864 INFO:teuthology.orchestra.run.smithi177.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30825.log.gz 2024-09-15T04:05:33.864 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29861.log 2024-09-15T04:05:33.864 INFO:teuthology.orchestra.run.smithi177.stderr: 88.8%/var/log/ceph/ceph-client.admin.33830.log: -- replaced with /var/log/ceph/ceph-client.admin.30944.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33521.log 2024-09-15T04:05:33.864 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T04:05:33.864 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31769.log 2024-09-15T04:05:33.864 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.29861.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33830.log.gz 2024-09-15T04:05:33.864 INFO:teuthology.orchestra.run.smithi177.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.29328.log.gz 2024-09-15T04:05:33.865 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33521.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-15T04:05:33.865 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31769.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33521.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31252.log 2024-09-15T04:05:33.865 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T04:05:33.865 INFO:teuthology.orchestra.run.smithi177.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29861.log.gz 2024-09-15T04:05:33.865 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31665.log 2024-09-15T04:05:33.865 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.3.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31769.log.gz 2024-09-15T04:05:33.865 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32386.log 2024-09-15T04:05:33.865 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31012.log 2024-09-15T04:05:33.865 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.32386.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31665.log.gz 2024-09-15T04:05:33.865 INFO:teuthology.orchestra.run.smithi177.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31252.log.gz 2024-09-15T04:05:33.865 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34106.log 2024-09-15T04:05:33.865 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33520.log 2024-09-15T04:05:33.865 INFO:teuthology.orchestra.run.smithi177.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32386.log.gz 2024-09-15T04:05:33.865 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.34106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30683.log 2024-09-15T04:05:33.866 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.33520.log: 89.0% 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34106.log.gz 2024-09-15T04:05:33.866 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31012.log.gz 2024-09-15T04:05:33.866 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31527.log 2024-09-15T04:05:33.866 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.30683.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33520.log.gz 2024-09-15T04:05:33.866 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-client.admin.31527.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30683.log.gz 2024-09-15T04:05:33.866 INFO:teuthology.orchestra.run.smithi177.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31527.log.gz 2024-09-15T04:05:33.866 INFO:teuthology.orchestra.run.smithi177.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-15T04:05:34.035 INFO:teuthology.orchestra.run.smithi110.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-15T04:05:34.131 INFO:teuthology.orchestra.run.smithi177.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-15T04:05:34.894 INFO:teuthology.orchestra.run.smithi110.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-15T04:05:34.946 INFO:teuthology.orchestra.run.smithi177.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-15T04:05:34.946 INFO:teuthology.orchestra.run.smithi110.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-15T04:05:34.983 INFO:teuthology.orchestra.run.smithi177.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-15T04:05:35.009 INFO:teuthology.orchestra.run.smithi110.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-15T04:05:35.011 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-09-15T04:05:35.011 INFO:teuthology.orchestra.run.smithi110.stderr:real 0m1.348s 2024-09-15T04:05:35.011 INFO:teuthology.orchestra.run.smithi110.stderr:user 0m4.178s 2024-09-15T04:05:35.011 INFO:teuthology.orchestra.run.smithi110.stderr:sys 0m0.238s 2024-09-15T04:05:35.053 INFO:teuthology.orchestra.run.smithi177.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-15T04:05:35.055 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-15T04:05:35.055 INFO:teuthology.orchestra.run.smithi177.stderr:real 0m1.356s 2024-09-15T04:05:35.055 INFO:teuthology.orchestra.run.smithi177.stderr:user 0m4.171s 2024-09-15T04:05:35.055 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m0.316s 2024-09-15T04:05:35.055 INFO:tasks.ceph:Archiving logs... 2024-09-15T04:05:35.056 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905333/remote/smithi110/log 2024-09-15T04:05:35.057 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T04:05:35.806 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905333/remote/smithi177/log 2024-09-15T04:05:35.807 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T04:05:36.635 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T04:05:36.645 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T04:05:36.645 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T04:05:36.648 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T04:05:36.736 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T04:05:36.758 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T04:05:36.758 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T04:05:36.761 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T04:05:36.776 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-09-15T04:05:36.778 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-09-15T04:05:36.781 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T04:05:36.781 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-09-15T04:05:36.781 INFO:teuthology.orchestra.run.smithi110.stdout:^* hv01.front.sepia.ceph.com 3 6 377 54 +226us[ +271us] +/- 43ms 2024-09-15T04:05:36.781 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 -24us[ -24us] +/- 59ms 2024-09-15T04:05:36.781 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 -281us[ -281us] +/- 67ms 2024-09-15T04:05:36.781 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T04:05:36.783 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T04:05:36.783 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-09-15T04:05:36.783 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv01.front.sepia.ceph.com 3 6 377 54 +242us[ +278us] +/- 43ms 2024-09-15T04:05:36.783 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 -8924ns[-8924ns] +/- 59ms 2024-09-15T04:05:36.783 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 -253us[ -253us] +/- 67ms 2024-09-15T04:05:36.783 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T04:05:36.783 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T04:05:36.792 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T04:05:36.792 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T04:05:36.806 DEBUG:teuthology.orchestra.run.smithi110:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T04:05:36.892 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T04:05:36.961 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T04:05:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T04:05:36.991 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T04:05:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T04:05:37.020 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T04:05:37.030 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T04:05:37.048 INFO:teuthology.task.internal:Duration was 803.478224 seconds 2024-09-15T04:05:37.049 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T04:05:37.065 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T04:05:37.065 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T04:05:37.069 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T04:05:37.107 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T04:05:37.110 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T04:05:37.501 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T04:05:37.501 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2024-09-15T04:05:37.502 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T04:05:37.563 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-09-15T04:05:37.564 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T04:05:37.594 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T04:05:37.594 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T04:05:37.606 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T04:05:37.662 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T04:05:37.662 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T04:05:37.675 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T04:05:37.804 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T04:05:37.814 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T04:05:37.815 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T04:05:37.840 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T04:05:37.865 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T04:05:37.878 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:05:37.883 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:05:37.904 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2024-09-15T04:05:37.928 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-09-15T04:05:37.970 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:05:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T04:05:38.003 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:05:38.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T04:05:38.026 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T04:05:38.036 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T04:05:38.040 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905333/remote/smithi110 2024-09-15T04:05:38.040 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T04:05:38.078 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905333/remote/smithi177 2024-09-15T04:05:38.079 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T04:05:38.113 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T04:05:38.113 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T04:05:38.116 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T04:05:38.162 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T04:05:38.177 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T04:05:38.177 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T04:05:38.192 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T04:05:38.192 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T04:05:38.196 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T04:05:38.208 INFO:teuthology.orchestra.run.smithi110.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 04:05 /home/ubuntu/cephtest 2024-09-15T04:05:38.217 INFO:teuthology.orchestra.run.smithi177.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 04:05 /home/ubuntu/cephtest 2024-09-15T04:05:38.218 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T04:05:38.231 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T04:05:38.258 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_latest} thrashers/sync workloads/pool-create-delete} duration: 803.4782235622406 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-15T04:05:38.258 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T04:05:38.347 INFO:teuthology.run:pass